With data scraping complete, moving on to analysis
Basic summation of projects, as proof of concept
Basic SAS program for importing CSV data and storing as time series data
Data pipeline error
Some (sub)total rows were being mistaken for actual hour entries.
Previously I cut off all elements ordered after the 'Hours Distribution
by Time Code' marker. This is insufficient because (1) some elements can
be slightly higher and (2) some elements can float to the previous page.
I've fixed (1) by fudging the y-dimension numbers. (2) appears to only
impact a single timesheet (2019-06-15). I've added some more debugging
to help diagnose and communicate this issue.
Adding exporters
Wrote and tested the long CSV exporter. Stubbed out the JSON exporter.
Fully functional timesheet parser.
The timesheet parser is a complete success. Some minor issues were
ironed out in the XML parser as well.
Next steps: writing to a time series database and beginning analysis.
Goodbye HTML, hello XML
Replaced HTML exporting/parsing with XML exporting/parsing. Also
replaced the 'high-level' function call with 'low-level' pdfminer
usage.
The XML parser handled validation and suppression of header/footer
content on its own.
From the PDF parser, XML is dumped to a file. From the XML parser, CSV
is dumped to a file. The new timesheet parser should read in that CSV
file.
Minor debug/toolchain update
Implemented time entry extraction; no assert errors!
There is still a major issue ahead of 'structured' data:
Hours data is leaking between entries. There are entries with no hours
at all. There are almost certainly some entries that have hours out of
order.
It will likely be necessary to re-sort all items ahead of processing
based on top then left style attributes. This is going to have the
consequence of invalidating some of the work I've already put into
parsing the data as-is.
Good luck, future me.
Started implementing time entry extraction.
Time entries are now being parsed and validated, though there are
numerous issues still to sort out.
I have a feeling that further development will require passing around
the `top` style attributes in the same way I'm passing around the `left`
style attributes. TBD though.
Significant updates
Wrote time sheet parser that ingests and validates all semi-structured
data. Next step is to interpret left styles as dates, so that hours can
be parsed into a time entry object.
Updated HTML parser to more completely filter out unhelpful data, and to
internally build the array of doubles (data and left style).
Toolchain upgrades
Migrated to `venv` for the Python module dependency(ies).
Added `README` to begin documenting the process.