merge_logs.py
Merge the split, overlapping log exports from ProfitView into one clean, de-duplicated, chronologically ordered log file, and keep that file up to date as you export more logs.
Why
ProfitView exports logs as a set of roughly 10 MB chunks:
profitview-2026-09-16-14-57-05-1.log profitview-2026-09-16-14-57-06-2.log profitview-2026-09-16-14-57-07-3.log
An export smaller than 10 MB is not split and has no chunk suffix, e.g. profitview-2026-11-06-13-08-40.log. Both forms are supported and can be mixed in the same folder.
That is awkward for three reasons:
- Exports overlap. ProfitView only keeps a limited log history, so each new export repeats a long stretch of entries you already have from the previous one.
- Chunks split entries. The 10 MB split can cut a multi-line entry in half, so the chunks cannot simply be deduplicated one by one.
- Timestamps shift across DST changes. The exporter shows old timestamps using the UTC offset in effect at export time. So the same event can show a different clock time, one hour apart, in exports taken in summer and in winter. A plain text comparison would then miss those duplicates.
merge_logs.py handles all of this and produces a single profitview-merged.log.
Requirements
- Python 3.8 or newer
- No third-party packages
Quick start
- Put
merge_logs.pyin the folder where you keep your logs. - Export the logs from ProfitView and drop the chunk files into that folder.
- Run:
python merge_logs.py
This creates profitview-merged.log.
Keeping the merged log up to date
This is the intended day-to-day workflow:
- Export the logs again whenever you like.
- Drop the new chunk files into the same folder, next to the existing
profitview-merged.log. - Run
python merge_logs.pyagain.
If profitview-merged.log already exists, the tool uses it as the base. It keeps everything already in it and appends only the entries that are genuinely new. After the merge you can delete the export chunks, but you don't have to: the tool recognises and skips entries it already has, so running it again with the same files present leaves the output unchanged.
The existing log is never edited in place. The new result is written to profitview-merged.log.tmp and only replaces the old file once it is complete. If a run crashes or is interrupted with Ctrl+C, your existing merged log is left untouched.
Usage
python merge_logs.py [inputs ...] [options]
inputs are log files and/or directories to scan. The default is the current directory.
| Option | Description |
|---|---|
-o, --output FILE | Output file (default: profitview-merged.log). If it already exists, it is used as the merge base. |
--split PERIOD | Write one file per month, quarter or year instead of a single file (default: none). See Splitting the output. |
--fresh | Ignore an existing output file and rebuild only from the export chunks. |
-n, --dry-run | Analyse and print the summary, but write nothing. |
--report FILE | Also write the summary to this file. |
--no-detect-shift | Skip DST/timezone shift detection. Faster, but misses duplicates across a DST change. |
-q, --quiet | Hide per-export output and progress bars. |
--no-progress | Hide progress bars only. |
Examples
# Default: merge everything in the current folder into profitview-merged.log python merge_logs.py # Check first what a merge would do python merge_logs.py --dry-run # One merged file per month: profitview-merged-2026-02.log, ... python merge_logs.py --split month # Collect exports from several folders python merge_logs.py . archive/2026-q2 # Rebuild the merged log from scratch (e.g. after adding older exports) python merge_logs.py --fresh # Write to a different file and save the summary python merge_logs.py logs/ -o bot-a.log --report bot-a-report.txt
Input files must follow the export naming scheme <prefix>-YYYY-MM-DD-HH-MM-SS-<chunk>.log, or <prefix>-YYYY-MM-DD-HH-MM-SS.log for an unsplit export. Other .log files are ignored, and the tool prints a note for each one it skips.
Splitting the output
With --split, the merged log is written as one file per period. The period is added to the output file name:
| --split | Example file name |
|---|---|
month | profitview-merged-2026-02.log |
quarter | profitview-merged-2026-q1.log |
year | profitview-merged-2026.log |
Pass the same --split option on every run. Existing split files are read back as the merge base, the same way a single profitview-merged.log is, and only the periods that get new entries actually change.
Some details:
- Any existing merged output is used as the base. This includes the single file and split files of every granularity. So you can switch granularity at any time without keeping the original exports: running
python merge_logs.py --split monthnext to an existingprofitview-merged.logsplits it into monthly files. Afterwards the summary lists earlier merged files that the run did not rewrite, such as the old single file. Delete them once you no longer need them. Until you do, every run reads them again as a base, which is harmless but slower. - No new exports needed. If the folder has no export chunks, the tool still runs as long as a merged output exists, so a plain re-split works.
- Files are contiguous. An entry goes into the file of its timestamp's period. The period never goes backwards, though: an entry whose clock is slightly earlier than the one before it stays in the current file. This happens with thread jitter, or when a DST seam moves the clock back across a month boundary. As a result, the split files concatenated in order are byte-for-byte identical to the single merged log.
- Month boundaries use the timestamps as written. Entries near a DST seam can be up to an hour off from the true time, so an entry from just after midnight on the 1st can end up in the previous month's file, or the other way round.
Example output
Using existing profitview-merged.log as merge base. Found 3 file(s) in 2 export set(s), 135.6MB total. Aligning 1/1 |████████████████████████████████████████| 100.0% 110.0MB done in 00:04 [ 1/2] profitview-merged.log 1 chunk(s) 2025-12-11 19:43 .. 2026-09-12 22:13 kept 349,323 dropped 0 [ 2/2] profitview-2026-09-16-14-57-05-1.log 3 chunk(s) 2026-06-18 15:40 .. 2026-09-16 14:20 kept 4,944 dropped 79,321 Merging |████████████████████████████████████████| 100.0% 135.6MB done in 00:07 Summary ------- input files : 3 in 2 export sets (135.6MB) merge base : profitview-merged.log, 349,323 entries carried over entries written : 354,267 duplicates removed: 79,321 overlap : 18.3% of parsed entries time range : 2025-12-11 19:43:15.556000 .. 2026-09-16 14:20:10.594000 output : profitview-merged.log (111.5MB)
Progress bars are only drawn when the output goes to a terminal, so redirected output stays clean.
How it works
- Group chunks into exports. A chunk numbered
1starts a new export. A file without a chunk suffix is a complete export on its own. The timestamp in the filename changes by a second or so from chunk to chunk, so the tool cannot use it to group chunks. - Order exports oldest first, by the first log entry each one actually contains. An existing merged log is treated as one more export, normally the oldest.
- Read each export as one continuous stream. The chunks are joined before parsing, so an entry split across two chunks stays whole. An entry is a timestamped line plus any following lines without a timestamp.
- Detect timezone shifts. Entries that appear only once near the start of each export are looked up in the previous export. The most common time difference, rounded to whole hours, is that export's shift.
- De-duplicate.
- The tool tracks the newest timestamp it has written so far, adjusted for any detected shift.
- Anything clearly older than that is a duplicate.
- Within a 5-second window around it, entries are compared by content instead. Log lines from concurrent threads can be slightly out of order, and this avoids dropping genuinely new ones.
- Write the entries byte for byte as they appear in the exports. The tool never changes entry text, line endings or invalid UTF-8.
Things to know
Coverage gaps
If there is a time gap between the end of one export and the start of the next, ProfitView discarded those entries before any export captured them. The summary shows a warning like this:
WARNING: 1 coverage gap(s) - entries purged before any export caught them: 2026-07-01 08:12:44 .. 2026-07-03 17:05:02 (2 days, 8:52:18) before profitview-...-1.log
The tool cannot fill such gaps; exporting more often avoids them.
Clock jumps at DST changes
Duplicates are detected using timestamps adjusted for the shift, but the output keeps each entry's original timestamp text. So where the merged log switches from an export taken in winter to one taken in summer (or the other way round), the timestamps can jump by an hour. The summary lists every detected shift.
Adding older exports later
Exports are always sorted by their contents, so adding an export older than the merged log still works. If you want a clean rebuild from only the chunk files, run with --fresh.
License
Released under the MIT License.