#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#

Linear Store issues:

Current/pending:
================
 Q-JIRA RHBZ     Description / Comments
 ------ -------  ----------------------
   5359 -        Linearstore: Implement new management schema and wire into store
   5360 -        Linearstore: Evaluate and rework logging to produce a consistent log output
*  5361 1145359  Linearstore: No tests for linearstore functionality currently exist
                   svn r.1564893 2014-02-05: Added tx-test-soak.sh
                   svn r.1564935 2014-02-05: Added license text to tx-test-soak.sh
                   svn r.1625283 2014-09-16: Basic python tests from legacystore ported over to linearstore
                   * No existing tests for linearstore:
                   ** Basic broker-level tests for txn and non-txn recovery
                   ** Store-level tests which check write boundary conditions
                   ** EFP tests, including file recovery, error management
                   ** Unit tests
                   ** Basic performance tests
   5464 -        [linearstore] Incompletely created journal files accumulate in EFP
   -    1088944  [Linearstore] store does not return all files to EFP after purging big queue <queue purge issue>
   -    1067480  [LinearStore] Provide a way to limit max count/size of empty files in EFP
   -    1067429  [LinearStore] last file from deleted queue is not moved to EFP  <queue delete issue>
   -    1067482  [LinearStore] Provide a way to preallocate empty pages in EFP
   6303 1180660  [linearstore] Roll back auto-upgrade of store directory structure
                   svn r.1652486 2015-01-16 Fix, mutually exclusive dir structure introduced
   5880  867826  [RFE] Qpid memory usage is not consistent across restart
                 Tools install
   



Fixed/closed:
=============
 Q-JIRA RHBZ     Description / Comments
 ------ -------  ----------------------
   5357 1052518  Linearstore: Empty file recycling not functional
                   svn r.1545563 2013-11-26: Propsed fix. VERIFIED
   5358 1052727  Linearstore: Checksums not implemented in record tail
                   svn r.1547601 2013-12-03: Propsed fix. NEEDINFO on algorithm
   5387 1036071  Linearstore: Segmentation fault when deleting queue
                   svn r.1547641 2013-12-03: Propsed fix. VERIFIED
   5388 1035802  Linearstore: Segmentation fault when recovering empty queue
                   svn r.1547921 2013-12-04: Propsed fix. VERIFIED
NO-JIRA -        Added missing Apache copyright/license text
                   svn r.1551304 2013-12-16: Propsed fix
   5425 1052445  Linearstore: Transaction Prepared List (TPL) fails with jexception 0x0402 AtomicCounter::addLimit() threw JERR_JNLF_FILEOFFSOVFL
                   svn r.1551361 2013-12-16: Proposed fix VERIFIED
   5442 1039949  Linearstore: Dtx recover test fails
                   svn r.1552772 2013-12-20: Proposed fix VERIFIED
   5444 1052775  Linearstore: Recovering from qpid-txtest fails with "Inconsistent TPL 2PC count" error message
                   svn r.1553148 2013-12-23: Proposed fix NEEDIFNO on reproduction and testing
   -    1038599  [LinearStore] Abort when deleting used queue after restart
                   CLOSED-NOTABUG 2014-01-06
   5460 1051097  [linearstore] Recovery of store which contains prepared but incomplete transactions results in message loss
                   svn r.1556892 2014-01-09: Proposed fix VERIFIED
   5473 1051924  [linearstore] Recovery of journal in which last logical file contains truncated record causes crash
                   svn r.1557620 2014-01-12: Proposed fix MODIFIED
   5483 -        [linearstore] Recovery of journal with partly written record fails with "JERR_JREC_BADRECTAIL: Invalid data record tail" error message
                   svn r.1558589 2014-01-15: Proposed fix
                   * May be linked to RHBZ 1039522 - VERIFIED
                   * May be linked to RHBZ 1039525 - VERIFIED
   -    1039522  Qpid crashes while recovering from linear store around apid::linearstore::journal::JournalFile::getFqFileName() including enq_rec::decode() threw JERR_JREC_BAD_RECTAIL
                   * Possible dup of 1039525
                   * May be fixed by QPID-5483 - waiting for needinfo, recommend rebuilding with QPID-5483 fix and re-testing. VERIFIED.
   -    1039525  Qpid crashes while recovering from linear store around apid::linearstore::journal::jexception::format including enq_rec::decode() threw JERR_JREC_BAD_REC_TAIL
                   * Possible dup of 1039522
                   * May be fixed by QPID-5483 - waiting for needinfo, recommend rebuilding with QPID-5483 fix and re-testing. VERIFIED.
   5487 1054448  [linearstore] Replace use of /dev/urandom with c random generator calls
                   svn r.1558913 2014-01-16: Proposed fix VEFIFIED
   5484 1035843  Slow performance for producers
                   svn r.1558592 2014-01-15 fixes an issue with using /dev/random as a source of random numbers for Journal serial numbers.
                   svn r.1558913 2014-01-16 replaces use of /dev/urandom with several calls to rand() to construct a 64-bit random number.
                   * Recommend rebuilding and testing for performance again with these two fixes. VERIFIED.
   5479 1053701  [linearstore] Using recovered store results in "JERR_JNLF_FILEOFFSOVFL: Attempted to increase submitted offset past file size. (JournalFile::submittedDblkCount)" error message
                   * Probability: 2 of 600 (0.3%) using tx-test-soak.sh
                   * Fixed by checkin for QPID-5480, no longer able to reproduce. VERIFIED
   5480 1053749  [linearstore] Recovery of store failure with "JERR_MAP_NOTFOUND: Key not found in map." error message
                   svn r.1564877 2014-02-05: Proposed fix
                   * Probability: 6 of 600 (1.0%) using tx-test-soak.sh
                   * If broker is started a second time after failure, it starts correctly and test completes ok.
                   * Problem: File is being recycled to EFP with still-locked enqueues in it (ie dequeued transactionally).
                   * Problem: Record alignment check writes filler records to wrong file when decoding bad record moves across a file boundary
   5603 1063700  [linearstore] broker restart fails under stress test
                   svn r.1574513 2014-03-05: Proposed fix. POST
                   * jexception 0x0701 RecoveryManager::readNextRemainingRecord() threw JERR_JREC_BADRECTAIL
   5607 1064181  [linearstore] Qpidd closes transactional client session&connection with async_dequeue() failed
                   svn r.1575009 2014-03-06: Proposed fix. POST
                   * jexception 0x010b LinearFileController::getCurrentSerial() threw JERR_NULL
   -    1064230  [linearstore] Qpidd linearstore recovery sometimes fail to recover messages with recoverMessages() failed
                   * jexception 0x0701 RecoveryManager::readNextRemainingRecord() threw JERR_JREC_BADRECTAIL
                   * possible dup of 1063700
   -    1036026  [LinearStore] Qpid linear store unable to create durable queue - framing-error: Queue <q-name>: create() failed: jexception 0x0000
                   * UNABLE TO REPRODUCE - but Frantizek has additional info
                   * Retested after checkin 1575009, problem solved. VERIFIED
   5651       -  [C++ broker] segfault in qpid::linearstore::journal::jdir::clear_dir when declaring durable queue
                   svn r.1582730 2014-03-28: Proposed fix by Pavel Moravec
                   * Bug introduced by r.1578899.
   5661       -  [linearstore] Set default cmake build to exclude linearstore
                   svn r.1584379 2014-04-03: Proposed solution.
                   * Run ccmake, select BUILD_LINEARSTORE to change its value to ON to build.
   5750 1078142  [linearstore] qpidd closes connection with (distributed) transactional client while checking previous transaction, broker signals error (closed by error: Queue Ve0-2: async_dequeue() failed: exception 0x0103 wmgr::get_events() threw JERR__AIO: AIO error)
                   svn r.1594215 2014-05-13: Proposed solution.
                   * jexception 0x0103 wmgr::get_events() threw JERR__AIO: AIO error. (AIO write operation failed: Invalid argument (-22) [pg=0 size=8192 offset=4096 fh=22])
   5655 1078937  [linearstore] Installation and tests for new store analysis tool qpid-qls-analyze
                   svn r.1596633 2014-05-21: Modified to run from installed location
   5767 1098118  [linearstore] broker segfaults when recovering journal file with damaged header
                   svn r.1596509 2014-05-21: Proposed solution (committed by pmoravec)
                   svn r.1599243 2014-06-02: Solution to additional case of file header corruption
   5924 1124906  [linearstore] Qpidd Will Not Start with Large Number of Queues
                   svn r.1614665 2014-07-30: Proposed solution
   5948 1121660  [AMQP 1.0] Broker restart failure with durable topic using non-durable exchange
                   svn r.1616287 2014-08-06: Proposed solution checked in by gsim
                   This turned out to be an AMQP error, fix does not affect store code.
   6043 1089652  [RFE]: Configuration option for linear store to delete or overwrite the used journal files.
                   svn r.1620426 2014-08-25: Proposed solution
   6147 1152012  [C++ broker linearstore] missing journal id in "trace Mgmt create journal." log
                   svn r.1631360 2014-10-13: Proposed solution
   6157 1150397  linearstore: segfault when 2 journals request new journal file from empty EFP
                   svn r.1632504 2014-10-17: Proposed solution by pmoravec   
   6230 1165200  [linearstore] qpid-qls-analyze fails when analyzing empty journal
                   svn r.1643053 2014-11-18: Proposed fix
   6232 1163650  [C++ broker] Linearstore segfaults when ulimit prevents creating new file in EFP
                   svn r.1640357 2014-11-14: Proposed fix by pmoravec
   6248 1167911  [linearstore] Symlink creation fails if store dir path is not absolute
                   svn r.1641689 2014-11-25: Proposed solution
   5671 1160367  [linearstore] Add ability to use disk partitions and select per-queue EFPs
                   svn r.1636598 2014-11-04: WIP: New EFP and journal dir structure using symlinks
                   svn r.1637985 2014-11-10: WIP: Auto-upgrade from old dir structure to new
                   svn r.1649081 2015-01-02: WIP: Specify new queue using qpid-config --durable together with --efp-partition-num and/or --efp-pool-file-size. Needs testing.
   -    1148807  [linearstore] Restarting broker with empty journal raises confusing warning
                   Fixed by svn r.1649081 of bug 5671 / 1160367 above
   6303 1180660  [linearstore] Roll back auto-upgrade of store directory structure
                   svn r.1652486 2015-01-16: Fix, mutually exclusive dir structure introduced
   5362 1145363  Linearstore: No store tools exist for examining the journals
                   svn r.1556888 2014-01-09: WIP checkin for linearstore version of qpid_qls_analyze. Needs testing and tidy-up.
                   svn r.1560530 2014-01-22: Bugfixes for qpid_qls_analyze
                   svn r.1561848 2014-01-27: Bugfixes and enhancements for qpid_qls_analyze
                   svn r.1564808 2014-02-05: Bugfixes and enhancements for qpid_qls_analyze
                   svn r.1578899 2014-03-18: Bugfixes and enhancements for qpid_qls_analyze
                   svn r.1583778 2014-04-01: Bugfix for qpid_qls_analyze
                   svn r.1652490 2015-01-16: Update for new partition capability and resulting dir structure change
                   * Store analysis and status
                   * Recovery/reading of message content
                   * Empty file pool status and management
   -    1066256  [LinearStore] changing efp size after using store broke the new durable nodes creation
                   Believed to be fixed by 5671 / 1160367
   6355 1188748  [linearstore] Installation error
                   svn r.1656771 2015-02-03: Proposed fix.
   6394       -  [linearstore] Make linearstore the default store for linux builds
                   svn r.1660430 2015-02-17: Proposed fix. 
                   svn r.1660702 2015-02-18: Corrected incorrect status messages.
   6399       -  Windows run_test.ps1 script needs revamp to match run_test
                   svn r.1671128 2015-04-03: Proposed fix for store tests
   6551 1223789  [C++ broker]: linearstore raising JERR_LFCR_SEQNUMNOTFOUND after sending many DTX transactions
        1223795           svn r.1680861 2015-05-21: Proposed fix.
        1224300
   6730 1261180  Make the journal flush timeout configurable in the C++ broker
                   svn r.1702130 2015-09-09: Proposed fix by Andrew Stitcher (astitcher)
NO-JIRA       -  Improve iostreams error reporting by printing out the value of errno
                   svn r.1702258 2015-09-10: Proposed fix
                   

Ordered checkin list:
=====================
In order to port the linearstore changes from trunk to a branch, the following svn checkins need to be ported in order:

no.   svn r  Q-JIRA     RHBZ       Date Alt Committer
--- ------- ------- -------- ---------- -------------
 1. 1545563    5357  1052518 2013-11-26 0.22-mrg
 2. 1547601    5358  1052727 2013-12-03 0.22-mrg
 3. 1547641    5387  1036071 2013-12-03 0.22-mrg
 4. 1547921    5388  1035802 2013-12-04 0.22-mrg
 5. 1551304 NO-JIRA        - 2013-12-16 0.22-mrg (aconway)
 6. 1551361    5425  1052445 2013-12-16 0.22-mrg
 7. 1552772    5442  1039949 2013-12-20 0.22-mrg
 8. 1553148    5444  1052775 2013-12-23 0.22-mrg
 9. 1556888    5362        - 2014-01-09
10. 1556892    5460  1051097 2014-01-09 0.22-mrg
11. 1557620    5473  1051924 2014-01-12 0.22-mrg
12. 1558589    5483        - 2014-01-15 0.22-mrg
13. 1558592    5484  1035843 2014-01-15 0.22-mrg
14. 1558913    5487  1054448 2014-01-16 0.22-mrg
15. 1560530    5362        - 2014-01-22
16. 1561848    5362        - 2014-01-27
17. 1564808    5362        - 2014-02-05
18. 1564877    5480  1053749 2014-02-05 0.22-mrg
19. 1564893    5361        - 2014-02-05
20. 1564935    5361        - 2014-02-05
21. 1574513    5603  1063700 2014-03-05 0.22-mrg
22. 1575009    5607  1064181 2014-03-06 0.22-mrg
23. 1578899    5362        - 2014-03-18 parts in 0.22-mrg
24. 1582730    5651        - 2014-03-28 0.22-mrg (pmoravec)
25. 1583778    5362        - 2014-04-01
26. 1584379    5661        - 2014-04-03
27. 1594215    5750  1078142 2014-05-13 0.22-mrg
28. 1596509    5767  1098118 2014-05-21 0.22-mrg (pmoravec)
29. 1596633 NO-JIRA  1078937 2014-05-21 (includes tools install update)
30. 1599243    5767  1098118 2014-06-02 0.22-mrg
31. 1599243    5767  1098118 2014-06-02
32. 1614665    5924  1124906 2014-07-30
33. 1620426    6043  1089652 2014-08-25
34. 1631360    6147  1152012 2014-10-13 (pmoravec)
35. 1632504    6157  1150397 2014-10-17 (pmoravec)
36. 1636598    5671  1160367 2014-11-04
37. 1637985    5671  1160367 2014-11-10
38. 1643053    6230  1165200 2014-11-18
39. 1640357    6232  1163650 2014-11-18 (pmoravec)
40. 1641689    6248  1167911 2014-11-25
41. 1642731 NO_JIRA        - 2014-12-01
42. 1649081    5671  1160367 2015-01-02
43. 1649082 NO-JIRA        - 2015-01-02
44. 1652486    6303  1180660 2015-01-16
45. 1656771    6355  1188748 2015-02-03
46. 1660430    6394        - 2015-02-17
47. 1660702    6394        - 2015-02-18
48. 1671128    6399        - 2015-04-03 (aconway)
49. 1680861    6551  1223789 2015-05-21 (pmoravec)
                     1223795
                     1224300
50. 1702130    6730  1261180 2015-09-09 (astitcher)
51. 1702258 NO-JIRA        - 2015-09-10
52.         NO-JIRA        - 2015-09-10

See above sections for details on these checkins.

Future work:
============
* One journal file lost when queue deleted. All files except for one are recycled back to the EFP.
* Complete exceptions - several exceptions thrown using jexception have no exception numbers
* Investigate ability of store to detect missing journal files, especially from logical end of a journal
* Investigate ability of store to handle file muddle-ups (ie journal files from EFP which are not zeroed or other journals)
* Look at improving the efficiency of recovery - right now the entire store is read once, and then each recovered record xid and data is read again

Code tidy-up
------------
* Remove old comments
* Use c++ cast templates instead of (xxx)y
* Member names: xxx_
* Rename classes, functions and variables to camel-case
* Add Doxygen docs to classes
* Make fid's consistent in name (fid, file_id, pfid) and format (hex vs decimal)
