SonarW Release Notes

V2.2 - Mar 01, 2018

  • Do not turn an existing collection to ingested without explicit override
  • Allow users to view and kill their own operations
  • Compression improvements
  • Exit if encryption key is changed
  • Cache the Cloud storage usage
  • Added cloud only collections
  • Added offlineStatus admin command
  • Added get_storage_usage admin command to monitor cloud and local storage usage
  • Cache serverStatus storage info
  • Display last relative DID in stats() and allow using it in {_id: {$did:[]}}
  • Throughout a query, subqueries are allowed in place of all arrays
  • Enable email sending on less restrictive environments (no-user,no-password)
  • Add iterGroup and headTail grouping
  • Add case insensitive grouping
  • Add $percentiles group accumulator
  • Show $comment in currentOp
  • Added $flattenToSet and $removeDuplicates
  • Add case insensitive to $similar
  • Added $regexSplit projections
  • Added $getHeadTail projection
  • Added $inTimePeriod projection
  • Added $inCIDR, $inNetwork, etc.
  • Added $comment agg step
  • Add support for $changeStream parsing and validation
  • Added “help” admin command
  • Added $caseEq comparison
  • Added jsonSchema support and validation
  • Added Join selector on other db
  • In join, added ability to project fields selected by a regex
  • Enable join between an element and an array that may contain it
  • Allow date type in $concat
  • Added new expression $tlike
  • Added $out syslog target and leef format type
  • Added syslog rfc5424 support in $out
  • Add $out in BSON format
  • support HDFS secondary server
  • Make a $bucket projection that works like $bucket aggregation
  • Improve $like to more resemble SQL “LIKE”
  • Fix update from last docid for purged collections
  • Write to system.authorization_failures the details of authorization failures
  • Fix bug in $contain with a word that start the same as another
  • Added impersonating facility through ismaster call
  • Append missing ids to unique ids sqlite file on start collection
  • Added command ‘rereadAuthCache’ to refresh auth cache

Mongo 3.6 feature support:

  • New “sessions”
  • Time zone in date operators
  • Aggregation - $currentOp
  • Aggregation - $listSessions
  • Aggregation - $listLocalSessions
  • Aggregation - REMOVE
  • Aggregation - $arrayToObject
  • Aggregation - $objectToArray
  • Aggregation - $mergeObjects
  • Aggregation - $dateFromString
  • Aggregation - $dateFromParts
  • Aggregation - $dateToParts
  • Aggregation - $lookup
  • Array Update Operator Enhancements - findAndModify
  • Array Update Operator Enhancements - findOneAndUpdate
  • Array Update Operator Enhancements - updateOne
  • Array Update Operator Enhancements - updateMany
  • Array Update Operator Enhancements - update
  • Array Update Operator Enhancements - bulkWrite
  • Array Update Operator Enhancements - Bulk.find.arrayFilters
  • Multi-Element Array Updates - $[] operator updates all elements in an array
  • Multi-Element Array Updates - filtered positional $[<identifier>] operator
  • Negative Array Index Position for push - $position
  • Wire protocol compression
  • authenticationRestrictions
  • Improve Login log-messages

V2.0 - Sept 15, 2017

  • Support storing data and managing Information Lifecycle on Cloud services: AWS S3, Google Cloud, Azure and local S3 repositories.
  • $project will not materialize until the projected document is needed.
  • If $gt+$lt are used together, use the sort table to check both of them at one.
  • Support on-disk encryption using KMIP or user supplied keys.
  • Support storing the unique _id index in multiple DB files rather than one.
  • Support $objectToArray,$arrayToObject
  • Cache results of expressions that are used multiple times on the same query
  • Optimize $group to build only the fields for $having, and only if $having passes, build the rest of the fields.
  • $contain, $like are available for $match/find() without $expr. If inside $expr, they are optimized out of it.
  • $regexp accepts additional parameter used as the options for the regex.
  • Automatically use top-k algorithm when sorting with a limit.
  • Admin command lmrm__metadata runs only once at a time.
  • All-document regex search (“/.*./”) speedups and ability to search numbers
  • Speedups when handling collections with a lot of deleted documents
  • Added $toJson, $fromJson projection operators
  • Block Integrity checking using signatures
  • Added $appendDid projection operator to find to add the documentID of the projected document.
  • Added $replaceAll projection operator
  • Added option to authenticate with SSL certificates
  • Added option to specify timezone in $dateToString
  • Added $similar to check similarity of two words (e.g. with soundex or distance )
  • Added $rangeTitle projection primitive
  • $expr with $gt, $lt etc. now use blocks
  • Add support for “undroppable” collections.
  • The Streams feature has been deprecated
  • Use a fixed set of works threads, with a minimum and maximum number of threads.

Mongo 3.4 feature support:

  • $facet aggregation stage
  • $graphLookup aggregation stage
  • $sortByCount aggregation stage
  • $count aggregation stage
  • $bucket and $bucketAuto aggregation stages
  • $replaceRoot aggregation stage
  • Support BSON_DECIMAL data type
  • $reduce
  • $switch
  • $split
  • $findChar
  • $zip
  • Views defined as aggregations
  • Collation
  • $addField
  • $bucket and $bucketAuto
  • $indexOfBytes
  • $indexOfCP
  • $indexOfArray
  • $indexOfArray
  • $range
  • $reverseArray
  • $indexOfBytes
  • $indexOfCP
  • $split (SonarW’s previous implementation of $split was changed to $splitMulti)
  • $strLenBytes
  • $strLenCP
  • $substrBytes
  • $substrCP
  • $isoDayWeek
  • $isoWeek
  • $isoWeekYear
  • $type (SonarW’s previous $type was changed to $isType)

V1.6 - Apr 10, 2017

  • Allow purge of part with a modified time that is the same as the purge time
  • Backup: Also save config file when doing backup
  • Do not start if not enough disk space for recovering ingestion file
  • Optimize $group with a lot of deleted documents
  • Optimization of $project with $lag and $skip
  • Improve memory management of aggregated values in $group
  • Speedup for $match results than have very fragmented results
  • Flush all collections during shutdown
  • Add permissions for root role to remove backup task
  • Fixes for database recovery: Rename collection, Recovery of a mid-recovery database
  • Add backup name to backup file list command
  • Database_Recovery: only delete transaction file after looking at all collections, remove empty recovery files, recover partially merged ingested collections
  • Fix possible deadlock when Logging in with unknown AUTH method on $external database (e.g. LDAP)
  • Fix ingest to create bounds for OID blocks
  • Fix various DLS and SLS bugs that occur when removing a database
  • Fix memory leak in bulk insert if one of the documents was a duplicate ID
  • Fix memory leak when dropping collection with a lot of deleted documents
  • Fix match with limit on an empty collection
  • Fix possible crash when killing an aggregation reading from a collection that was generated by $out before the generating aggregation was finished
  • Add “undroppable” collections, and admin command to set a collection as “undroppable”
  • Fix crash in geospatial commands
  • Do not reuse collection IDs even if they have been removed
  • Add table_constraints field to the information_schema collection
  • Projection speedup for flat documents
  • Fix group-sort-limit when there are a lot of groups, limit taken at the wrong time
  • Add TCP keepalive to avoid open connection forever
  • Avoid using the IEEE negative zero in output
  • Support $in, $nin for objects in array of objects
  • Fix crash is reporting the last bad duplicate ID
  • Fixed Star Projection ( ‘$dim.$*’:1) in $join to project correct document
  • Assign a UUID to each collection
  • Fix possible toString crash on long floating point numbers
  • $out to preserve collection definitions (e.g. auto-fields) if already exists
  • Support $group $sum when result is more than 32 bit but added elements are 32 bit
  • Fixed projection of sub-objects in sub-arrays for MongoDB compatibility
  • Added $addFields projection
  • Fixed deadlock on large number of concurrent drops and creates of same collection
  • Optimized logic for choosing whether or not to use disk-based sort

V1.4 - Dec 5, 2016

  • $user as a projection operator (currently logged-in user)
  • Geospatial - full geospational support, geo indexes (including multi), $geoWithin, $geoIntersects, $nearSphere etc.
  • Support join conditions using geoWithin
  • Admin command to find invalid geo shapes in collection that could not be indexed
  • Text search - compatible with MongoDB as well as $tsearch extension
  • Full compatibility with MongoDB 3.2
  • $lag projection operator
  • $headTail projection
  • $struct projection
  • $objectId projection to make a new _id
  • $like projection
  • $switch projection
  • $zip projection
  • $reduce projection
  • $indexOf projection
  • $diff group operator
  • $diffFirst group operator
  • $firstDoc and $lastDoc group operator
  • $bit update operator
  • Sending Email for important HADR events
  • Backtrace admin command to generate gdb strack trace
  • Add sleep, _mergeAuthzCollections, repair_cursor, geoNear, list indexes, cloneCollection, killCursors and hashBsonElement admin commands
  • Add pivoting to $addToSet group operator
  • Add document validation support
  • Support IPv6
  • Add $redact aggregation step
  • Upgrade to new V8 version
  • Make $query return empty array if nothing comes from query or column does not exist
  • Data-Level Security (DLS) and Schema-Level Security(SLS)
  • $out to S3
  • $out to file can generate parts and zipped individually
  • $out waits for any other $out to the same collection to finish before starting
  • Support upsert in $out
  • Option to disable javascript , —-noscripting
  • Support unicode column names
  • Support for auto-managed creation and last update timestamp per documents in a collection
  • Recompression based on age of data
  • Deprecate $between
  • Support INFORMATION_SCHEMA (for SonarSQL)
  • Support $graphLookup (from MongoDB 3.4)
  • Support $sample aggregation step
  • Extensions to $js
  • Bitwise expressions - bitsAllClear, bitsAnyClear, bitsAllSet, bitsAnySet
  • Join DIM table can be based on a subquery

V1.2.2 - Jun 9, 2016

  • Views
  • Streams
  • Window functions
  • Join with collections in another database
  • Aggregation pipelines in $in/$nin/$all subqueries
  • $out to S3 buckets
  • $diff
  • $diff within joins using $$CURRENT and $* project
  • $query with * projection
  • localhost_only role
  • ssh key-based logins
  • Union as an aggregation step
  • Join can use a subquery as a DIM

V1.1.12 - Apr 11, 2016

  • Ability to query versions of documents using the auto-update columns ($updates_since, $updates_until)
  • Faster group+group and join+group on very large results
  • Ability to do an update with a $expr
  • $did projection operator
  • $orphan capability in joins
  • Cache DIM *:1 projections
  • Ability to use % as a wildcard in regex with the ‘m’ option
  • $strict aggregation option (check or not check column existence)
  • Optimize $project+$sort
  • $var, $varPop,$stddev, $stddevPop window functions
  • More sophisticated $did projection operator - allowing iteration on collections from the Studio
  • $let projection primitive
  • Non materialized join if before $window
  • $toRegex
  • Add ability for $regex to work with array of regexes, matching one of them
  • DB Links for pipelines and union steps

V1.1.11 - March 14, 2016

  • StdDev for group
  • Improvements to disk sort when SSDs are available
  • Speed-up for sort single column
  • Recompress option for older data (slower but better compression)
  • Support for connection from MongoDB Scout
  • Support for normal find syntax for $having and others
  • Auto-timestamp fields ( Automatic last-update and insert-time columns)
  • Metadata compute speedup
  • New HADR method including using ZooKeeper
  • Improved recovery procedure
  • SCRAM as default authentication method
  • Improved cursor release (LRU)
  • Speed improvements on insert and ingest
  • MaxTimeMS does not affect cursors past first result
  • Document versioning and timestamps
  • $map and $arrayReduce
  • DBref traversal/de-ref projection primitive, including .*
  • Refuse inserts if there is no more disk space
  • Support RHEL6
  • Option to turn off compression completely
  • listCursors admin command
  • asserts() print a message on prod version
  • Recovery: Recover entire collection at once, save pre-recovery state of the collection
  • Allow setting the max size of one field in a disk-based sort
  • db.stream.<stream name> was changed to db.<stream name>
  • Added support for subqueries on views
  • If a block is used multiple times, only read it once.
  • Revive and improve SonarCSV
  • Enable passing conf file parameter on the command line

V1.1.4 - Feb 5, 2016

  • SonarW will now have a default user if no user is present
  • $contains and containsTuple speedups
  • Date function implementation (for enabling SoanrSQL translation)
  • $toDateTime and $toDate
  • Added $addMonth
  • Support for Amazon Linux
  • Add $having to $group and $join , using regular find() syntax
  • Added support for $warning
  • lmrm__metadata command to v8, and db.getCollectionNames

V1.1 - Jan 15, 2016

  • Improvements to pipeline optimizer
  • Fix slowness is sort if most data is not selected
  • Snippets: Make sure snippet is not partly in one doc and partlyin another
  • Warehouse Management Service - monitoring and statistics
  • SOT Dashboard
  • Redeem
  • Do not let the oplog start of an old SOT agent is still alive
  • Kill SOT agents when parent processes dies
  • $arrayElemAt
  • $elementAt
  • $concatArrays
  • $flatten
  • Added 128bit hashing functions for distinct docs and objects
  • Fixes for js map-reduce
  • Support for $lookup
  • min/max speedup
  • Added $sonarSyntax option to find()
  • Added $containa , can work with a string and an array
  • Added $containsTuple
  • Added $query projection operator
  • Improvements to disk sort
  • Added allow_duplicate_ids admin command
  • Added ‘journal’ file in collection dir to recover deletes and inserts that did not persist
  • Change $concat treats nulls as empty strings
  • Automatic document signature column
  • Support multi-join an rework join syntax
  • Join speedups
  • When ingesting, merging into existing part if possible
  • Cache the headers in one file
  • Grouping lookaside cache
  • Man command is now caseless
  • Support elemMatch in find
  • Support elemMatch in one unwind
  • Added $distinctDocs as an aggregations step
  • Recover uningested BSONs from ingest queue after crash
  • Added $hasPrivilege projection operator
  • killOp and currentOp per-user
  • Added pivot ability to all the $group function
  • Added the “memory_stats” admin command
  • Convert projection primitives to constant if all their inputs are constant
  • Add ‘*’ to {$project { “*”:1 } } and in $join’s $project
  • Cost based optimization: Evaluate expressions that can use the header only, then simple expressions, then regex, then $expr
  • Added $setParameter and $getParameter

V1.0 - Oct 2, 2015

  • First GA release of SonarW