Make the sleep duration dynamic to account for the time spent in
loop_step.
This improves responsiveness when repeatedly consuming newly
arriving docs.
Use float epoch seconds (time.time()) as the time type for
MailFetcher.last_checked to allow for natural time arithmetic.
Renamings:
loop -> loop_step
delta -> next_mail_time (this variable names a point in time, not a duration)
Extracting the 'loop' fn is a preparation for later commits where a
second type of loop is added.
1. Store the mtime of ignored files so that we can reconsider them if
they have changed.
2. Regularly reset the ignore list to files that still exist in the
consumption dir. Previously, the list could grow indefinitely.
Previously, the second mtime check for new files usually happened right
after the first one, which could have caused consumption of docs that
were still being modified.
We're now waiting for at least FILES_MIN_UNMODIFIED_DURATION (0.5s).
This also cleans up the logic by eliminating the consumer.stats attribute
and the weird double call to consumer.run().
Additionally, this a fixes memory leak in consumer.stats where paths could be
added but never removed if the corresponding files disappeared from
the consumer dir before being considered ready.
Check list length before accessing the first two elements of
'dispositions'.
The list may have only a single element ('inline') or may be empty in
mailformed emails.
Previously, all errors raised during mail fetching were silently caught
and printed without backtrace.
To increase robustness and ease debugging, we now fail with a backtrace
on unexpected errors.
LogEntry.action_time expects a Django timezone object instead of a builtin datetime.
This fixes a runtime warning of the following kind:
RuntimeWarning: DateTimeField LogEntry.action_time received a naive datetime (2018-03-28 20:53:01.714173) while time zone support is active.
