1.6.522 Oct 2017 18:00
Another portion of tokenization fixes. Fix memory leak in spf caching logic. Fix milter commands pipelining. Fix newlines detection. Filter nan and inf when adding scores. Implement headers flags in mime parser. Support Expires header when using HTTP maps. Actively load skip hashes map in fuzzy storage. Add workaround for IPv6 in sendmail. Authentication Results: Fix SPF smtp.mail_from. Check for magic when checking for an archive. Deal with another case when processing exceptions. Deal with URLs with no slashes after protocol. Do not allow garbadge when checking url domain. Do not ignore short words. Do not strip last character in the last word. Do not treat script content as text. Erase unknown HTML entities. Fix another tokenization issue. Fix DKIM forgeries via multiple headers. Fix emails detection. Fix empty threshold check in greylisting module. Fix enormous scores for R_WHITE_ON_WHITE. Fix loading of per-user redis backend for statistics. Fix multiple headers in DKIM headers list. Fix obscured url in format user@@example.com. Further tokenization fixes. Load skip map from all processes as shared cache is unavailable. Lowercase words. Milter headers: skip_local / skip_authenticated settings. Milter headers: X-Spamd-Result header if X-Virus ran first. Ratelimit: fix whitelisted_rcpts matching. Some more fixes towards emails detection. SpamAssassin: Fail check_freemail_header if regexp didn't match. Use greylisting threshold in greylisting module.
1.6.410 Sep 2017 11:36
Add method to get all content-type attributes in Lua. Add some sanity checks for actions and controller. Allow randomly select User-Agent from a list. Deal with obscured URLs with @ symbols. Milter headers: support adding/removing arbitrary headers from config. Add another workaround to display history properly. Add missing rspamadm control options to help. Auth-Results: Multiple DKIM signatures. Crash in URL processing. Default monitoring domain for surbl plugin. Detection of maillist optimized and fixed. Do not cache SPF records with PTR elements. Fix blacklists and DMARC in whitelist. Fix exceptions list in surbl. Fix processing of closed tags. Fix PTR processing in SPF. Lowercase HTTP headers to make them searchable from Lua. Fix options.local_networks setting. Ratelimit: lowercase email addresses. Rebalance and slightly rework MX check plugin. Fix Redis script loading in DMARC; URL tags; URL reputation. Reject invalid bh for DKIM signatures earlier. Remove incorrect method `task:set_metric_subject`. Fix rewriting subjects via force actions module. Fix RPM postinstall. Treat 'rewrite subject' as spam action. Try harder to find urls. Use full URL when making an HTTP request. Use raw urls when sending requests to redirector. Use weight from map for fuzzy scoring. Penalise R_BAD_CTE_7BIT for utf8 messages.
1.6.327 Jul 2017 08:13
Fix semicolons parsing in the content type. Add EBL to the default config. Allow to configure monitored. Allow to skip specific hashes in fuzzy storage. Multimap: checking of symbol options. Redis settings: support checking multiple keys. ARC: Fix Lua 5.3 compatibility; timestamp should be integer. Avoid changing content-transfer-encoding header's value. Don't use whitelist/greylist maps as regexp, but as map. Fix get_content method. Header checks: Fix get_raw_header method. Header checks: REPLYTO_UNPARSEABLE rule. Lua_http: freeing. Milter headers: custom headers: removing headers. Parse HREF urls without explicit prefix. WHITE_ON_WHITE: Ensure score is matched to part that fired the rule. Escape strings inside HTML in history.
1.6.208 Jul 2017 09:46
Remove Rambler email bl for now. Switch RAMBLER_URIBL to a locally managed source. Switch from ragel to C for Content-Type parsing. Add `-e` option for lua_repl. Add per-domain emails normalisation rules. Add sessions cache to debug dangling sessions. Add short_text_direct_hash for fuzzy check module. Add text_part:get_stats function. Allow to add custom processing script for surbl. Allow to check reply-to email. Allow to customize spam header, remove existing spam headers. Allow to disable specific workers in the config. Allow to discard messages instead of rejection. Allow to specify custom delimiter in emails plugin. Allow to specify custom User-Agent for rspamc. Allow to store symbols data in Clickhouse. Allow to use HTTPS when connecting to Clickhouse. Enable sessions cache tracking for milter connections. Implement per-line mode in lua_repl (like `perl -p`). Implement rdns-curve plugin based on rspamd cryptobox. Improve maps cached data lifetime. Improve maps checking frequency. Improve monitored timeouts logic. milter_headers: add `extended_headers_rcpt` option. Milter headers: Add X-Spam-Flag to rmilter-compatibility headers. Milter headers: remove-header routine. Multimap: received filters for extracting TLDs from hostnames. Normalize email aliases in emails module. Re-add rambler email bl (as hashed list). Reload file maps more frequently. Rework newlines strip parser one more time. Skip updates for messages scanned via controller. Split long DKIM public keys. Store more data when stripping newlines. Support SPF macros transformations. Support suppressing DMARC reports for some domains. Add missing `break` statement. Allow modifiers in SPF macros. DKIM sign tools: edge-cases around use_esld. Do not cache SPF records with macros. Do not overwrite score when setting pre-action. Fix comparision logic. Fix DKIM base64 folding for milter flagged messages. Fix emails module configuration. Fix folding for arc headers when milter interface is used. Fix gmail dots.
1.6.114 Jun 2017 14:53
Allow to init resolver without rspamd_config. Do not crash when resolver failed to initialize. Fix abstract context layout. Fix CGP helper reply parsing. Fix crashes when socket write errors occur. Fix parsing IPv6 nameservers in resolv.conf. Milter: Don't defer on "greylist" action.
1.6.012 Jun 2017 13:58
Milter protocol support. ARC support (signing validation). New statistics model for Redis storage. New expiration algorithm for internal caches. DMARC reporting support. Spamtrap plugin. URL redirector improvements. Compression support in proxy. Various performance improvements. Miscellaneous changes/fixes. https://rspamd.com/announce/2017/06/12/rspamd-1.6.0.html
1.5.901 Jun 2017 21:10
Increase min_bytes to avoid FP. Remove ratelimits from default configuration. Fix accepting on IPv6 sockets. Zero fill sockaddr_un. Add `text_multiplier` param. Check min_bytes for images as well. Do not add exact hashes from different parts. Fix memory leak when accepting from unix sockets. Fix some more issues about duplicated fuzzy requests. Phishing: fixed strict_domains. Skip text parts when checking binary parts in fuzzy check. Add the same duplicates protection for all fuzzy hashes types. SPOOF_DISPLAY_NAME: Use all SMTP/MIME recipients; validate assumed spoofed display name domains to contain a dot.
1.5.823 May 2017 18:18
Fix memory leak in fuzzy check. Fix memory leak in maps scheduling. Multimap: email:domain:tld filter. DKIM Signing: avoid nil index when From header is missing. Do not set bayes probability if we don't use it. Do not stop on illegal unicode points - replace them. Fix brain-damaged spamc protocol for now. Fix Redis timeout set. Fix spamc support in rspamd proxy. Fix Multimap Received IP filters with Redis. Parse rgb a (x,x,x ,x ) css colors. Reresolve upstreams even if there is a single server there. Treat empty address as valid. Try harder to detect CTE. Try to deal with v4 mapped to v6 addresses on accept. Add `wsf` and `hta` bad archive extensions. Fix configuration option. Fix result parsing for SAVAPI. Logging improvements. Prevent MID_CONTAINS_FROM from firing on empty address. Reduce digit- number transmission penalty. Relax CTYPE_MISSING_DISPOSITION rule.
1.5.702 May 2017 13:34
Fix corruption when multiple fuzzy are defined. Fix learn condition in fuzzy check. Add rules to detect bad 8bit characters in From and To. DKIM signing: sign_networks/local address specific use_domain settings. Support numeric arguments for Redis requests. Deal with 8bit characters in email addresses. Fix couple of issues in FORWARDED rule. Fix passing data to log helper when many symbols defined. Fix R_BAD_CTE_7BIT rule. Fix REPLYTO_UNPARSEABLE rule. Fix setting of email address. Rspamadm grep: Disable Lua patterns in string search by default. Add Lua 5.3 workaround. Add lua methods to detect if a part has 8bit characters. Allow session-less lua dns requests. Allow to append greylist end time to message reported. Avoid `nil` table. Disable dkim_signing if redis is specified but not configured. Fix build with pcre2. Fix rule. Fix warnings. Format floating point number. Push email flags to the lua API. Silence some warnings. Silence warning. Try all hostname regexps to find the most significant one. Fix add header filter in history.
1.5.619 Apr 2017 17:29
Add unigramms support in bayes. Allow configurable sign headers for DKIM. Allow to add unigramm metatokens from Lua. DKIM Signing: envelope match exception for local IPs. Always try to adjust filename. Do extra copy to ensure that original content is never touched. Fix SPOOF_REPLYTO rule. Ignore Rmilter added Received. More fixes for hashed email dnsbls. Plug memory leak in chartable module. Display multiple alerts at once in WebUI.
1.5.510 Apr 2017 10:32
Fix classifier learning with Redis backend. Fix issue when parsing encoded rfc822/messages. Add escaped version of lua_ucl import. Add task:headers_foreach function. Allow to process filenames from content type. Allow to query hashed emails. Ignore bayes with mostly metatokens or with too few text. Probabilistically skip metatokens. Retrieve all virus names from SAVAPI. Rework classifiers lua metatokens. Store headers order. Store text tokens inside bayes tokens. Use cached shingles keys. Add missing score normalisation for HFILTER_URL_ONLY. Avoid lookup in absent hash. Check return values from Lua functions called from C. Do not count sending and loading time in rspamc. Escape json strings for controller rejplies from Lua. Fix archive scans for savapi. Fix domain_only emails RBL. Fix ip_score map configuration. Fix JSON output for history_redis. Fix one character length substrings search. Fix parsing of non-RFC compatible Exim received. Fix parsing of options for workers with the same type. Fix processing of small tokens vectors. Fix rfc2047 tokenization. Fix typo. More fixes for inplace decoding. Try to avoid modifications of the original data. URL redirector: Fix call to is_redirector. Set token data as uint64_t instead of chars array. Check if neighbours' history backend versions match. Disable phrase connectors replacement in history filtering. Disable phrase connectors replacement in symbols filtering. Do not hide messages with bad subject, just replace it with '???'. Fix error message. Fix history v2 display. Fix legacy history. history: break To address lists on commas. Increase default timeout to 20 seconds. Save some history table space.
1.5.428 Mar 2017 15:57
Add history_redis default configuration. Add spoofed rules. Add URL_IN_SUBJECT rule. Allow to get task's subject. Allow to specify maximum number of shots for symbols. Distinguish URLs found in Subject. Memoize LPEG grammars. Parse else parts in SA rules. Process subject for mixed characters. Resolve url chains in url_redirector module. Stat greylisted messages as greylisted not soft-rejected. Support checking for redirector in Lua SURBL. Support tag_exists SA function. Work with broken rfc2047 tokens. Check all watcher's dependencies. Do not compile hyperscan with no SSSE3 support. Do not crash if cannot decode qp encoded part. Fix dependencies of DKIM when multiple signatures are found. Fix lists in whitelist plugin. Fix one-shot symbols weight calculations. Fix options and shots match. Fix order of symbol options. Fix parsing of dot at the end of the address. Fix parsing of lua table arguments. Fix processing of subject words. Fix string split memoization. Fix templates grammar usage. Fix various issues related to Lua stack manipulation. Force actions: Use postfilter if we have honor_action / require_action. Further fixes to avoid PHISHING FP. Preserve order of options in symbols. Rspamadm grep: deal with unusually-formatted logs. Use hostname suffix when dealing with history. Remove outdated SA rules. Add flexible columns. Add footable. Add sender, recipients and subject columns. Allow message-id break. Fix history clustering. Fix history display. Fix sorting. Humanize sizes. Initial move towards footable. Remove datatables. Replace `.values` method with `.map`. Rework v2 symbols display. Try to normalize frequencies. Unbreak WebUI. Use Footable to draw Throughput summary table.
1.5.317 Mar 2017 14:05
Add composite for hacked wordpress phishing. Fix base64 decoding when there are unparseable characters. Additional symbol metadata in metadata exporter. Add method to get protocol reply from Lua. Add symbols when tagged rcpt/sender are normalised. Add task:get_symbols_all() function. Allow multiple formats of DKIM signing key. Allow to cache and use flexible protocol reply. Allow to set one_shot flag from register_symbol. Allow to skip certain types of hashes when learning fuzzy. Cache and insert scan time into the protocol. Detect newlines in rspamc --mime. DKIM signing: support use of maps. Greylist: Support excluding low-scoring messages from greylisting. Implement lua history in controller. Implement redis history querying. Preliminary implementation of redis history plugin. Support using request headers in settings. Change default template to deal with non-ASCII characters. Deal with lists of maps in whitelist module. DKIM signing: use domain-specific signing key. Do not reallocate completed zstd buffer. Do not use local_addrs in proxy. Fix crash when resolver is undefined. Fix double free when closing lua_tcp connections. Fix for lua 5.3. Fix freeing of arrays iterators. Fix issue with task:get_symbol and symbols with no metric. Fix log line duplication in `rspamadm grep`. Fix memory corruption on termination. Fix out-of-bound access in base64 decode. Fix ratelimit + greylisting. Fix subject rewriting. Fix task:set_recipients function. Fix URI_COUNT_ODD rule. Follow the traditional symbols conventions in RCPT_COUNT rule. Greylist: Suppress greylist action for whitelisted hosts too. Metadata exporter: use rule-specific settings for emails. Properly set missing fields in exporter. Proxy: max_retries option. RCPT_COUNT fixes. Rework HAS_X_PRIO rule to match symbols conventions. Update issues in ac-trie. Use optimised base64 decoding in DKIM. Add preliminary v2 history parser. Allow different history parsers. Display symbols. Rework history v2 function.
1.5.207 Mar 2017 16:49
Antivirus: use scanner-specific redis prefix. Couple of fixes for DKIM signing module. Distinguish missing and broken mandatory headers. Do more heuristical detection for missing CTE. Do not resort cache on each check. Fix CGP escaping. Fix MISSING_MIME_VERSION rule for plain messages. Fix parsing of cte in expressions. Fix partial matches in rspamadm grep. Fix setting class on style field. Add default config for spamassasssin plugin. Add default configuration for antivirus module. Add dkim signing docs. Add mx_check default config. Add replies config. Add trie default config.
Add heuristic to find text parts in files. Add rule to detect broken content type. Allow to extract CTE in Lua API.
Allow to set from address for a lua_task. Allow to set recipients of a task from Lua. Enchance text_part:get_content method. Remove + aliases from emails. Support rmilter block and dkim signature in CGP helper. Support running event loop from Lua. Auto-switch Throughput units to `msg/min` for very low rate. Update D3Evolution to 0.0.2.
1.5.102 Mar 2017 17:55
Fixed processing order for postfilters. Fixed DKIM signing plugin. Fixed ClamAV support in antivirus module.
1.5.001 Mar 2017 15:33
New MIME parser. Switching from libiconv to libicu. WebUI rework. Lua TCP module rework. URL redirector module. Rmilter headers module. DKIM signing module. Force actions module. Reworked improved metadata exporter. URL tags plugin. URL reputation plugin. Multimap received maps. Changes in RBL module. Support for Avira SAVAPI in antivirus module. Neural net plugin improvements. Fuzzy matching for images. New rules. Rspamadm grep. Performance improvements. Stability improvements and bug fixes. https://rspamd.com/announce/2017/03/01/rspamd-1.5.0.html
1.4.416 Feb 2017 11:24
Fix bad memory leak in TLS certificates validation. Fix memory leak in HTTP maps. Replies module: fix symbol weight. Add missing macro. Do not treat IDN urls as bad URLs. Fix build with gcc 4.2. Fix cast. One more definition missed on Linux.
1.4.318 Jan 2017 14:29
Plugged memory descriptor leaks; fixed race conditions other issues.
1.4.206 Jan 2017 14:32
Fixes for DKIM parsing signing, fuzzy redis backend, dependencies in SpamAssassin plugin, URL detection, trie, fann_redis a memory leak. Added toggle for disabling SURBLs. New rules. Other features/bugfixes.
1.4.130 Nov 2016 15:14
Clickhouse plugin. Support for embedding maps in config. Lua modules debugging improvements. New rules. Multiple fixes to the ANN module. Other bugfixes. https://rspamd.com/announce/2016/11/30/rspamd-1.4.1.html
1.4.021 Nov 2016 12:55
Redis pool support
New neural nets plugin. Neural nets plugin has been reworked to store both training vectors and neural nets in Redis. Bayes improvements. New Antivirus plugin. New MX check plugin. Compression support in the protocol. Reworked model for DNS failures in SPF, DKIM and DMARC. Adaptive user-defined ratelimits. Monitored objects. Redis backend for fuzzy storage. Delhash support for fuzzy storage. Metric exporter and metadata exporter. Dynamic configuration in redis. Users settings in Redis. Errors ring buffer. Messages rework. Multiple updates to Rspamd Lua API. Rules improvements. WebUI improvements. Bugfixes. https://rspamd.com/announce/2016/11/21/rspamd-1.4.0.html
1.3.501 Sep 2016 12:41
Fix issue with finding of end of lines pointers. Add DMARC_NA / R_DKIM_NA / R_SPF_NA / AUTH_NA symbols. Add spamhaus DROP dnsbl. Add termination callbacks for workers. Add user-agent for rspamc. Allow parsing of mailbox messages from the commandline. Support excluding selected users from ratelimits Yield DMARC_DNSFAIL on lookup failure. Banish table.maxn from Lua parts. Do not use headers to calculate messages digests Fix absurd scores for HFILTER_URL_ONLY. Fix actrie patterns. Fix format string. Fix handling of ' 0' in lua_tcp. Fix issue with empty messages and dkim. Fix min_learns option. Fix options for SPF dnsfail symbol. Fix processing of symbols when reject limit is reached. Settings: fix `authenticated` parameter (#886). Try to avoid race condition when using rrd. Use the correct macro to get the size of controller. Fix build and tests on OpenBSD.
1.3.423 Aug 2016 12:41
Rspamd reload command has been fixed. Better ASN/country support. Variable maps in the multimap module. DNNSEC stub resolver support. DMARC and DKIM module fixes. Statistics backend now uses global redis settings. Tasks checksums. DKIM signature header is now folded by Rspamd.
MIME messages. Ratelimit module fixed. Fixed X-Forwarded-For header processing. https://rspamd.com/announce/2016/08/22/rspamd-1.3.4.html
1.3.315 Aug 2016 14:12
Important bugfix for fuzzy hashes (please update). Support for redis maps in multimap plugin. Important fix for hyperscan cache. SARBL URIBL added to default configuration. Lua API fixes and improvements. https://rspamd.com/announce/2016/08/15/rspamd-1.3.3.html
1.3.208 Aug 2016 17:06
Fixed multi value header processing. Attributes in HTML tags are now HTML decoded. Fixed processing of last element of DMARC record. Reworked Hfilter module. New SPF plugin features. Better HTML support. Improved settings matches for authorized users. Redis integration fix. DKIM signing fixes improvements. URL detection fixes.
1.3.101 Aug 2016 17:42
Removed systemd socket activation. New features in multimap: conditional maps; support for multiple symbols and scores per map; hostname map type. Bugfixes for greylist module. Fixed Catena password scheme. Fixed writing of Hyperscan cache. Implemented message size limit. Fixed configuration files includes logic. Other bugfixes and improvements.
1.3.025 Jul 2016 14:30
Proxy Support. Fuzzy storage mirroring. Redis replication support. Improved content filtering. Internal greylisting supporting. Replies module. DKIM signing support. WebUI improvements. Many other changes that improve stability, filtering quality and performance. Bug fixes to URL detection logic and phishing detection. The chartable module has been completely rewritten to provide more useful homograph detection.
1.2.820 Jun 2016 11:10
Another fix for exim workaround (#637). Fix unencrypted passwords processing in the controller. Fix setting path for lua (#652). Fix usage of rdns reply structure (#654). Use file lock in logger to avoid deadlocks. Add `application/octet-stream` mime type for `pdf` extension (by @moisseev). Implement new automata to skip empty lines for dkim signing (#651). Fix parsing of missing classes. Clarify some rspamc arguments (by @fatalbanana). Correct suppress spelling.
1.2.725 May 2016 16:31
Slightly reduce weights of rules with high FP rate. Add workaround for rspamd-1.3. Fix possible FP in TRACKER_ID rule. Simplify MISSING_MIMEOLE rule. Add workaround for gmime CTE stupidity. Fix mime headers processing. Fix false positive URL detections in text parts. Fix Exim shutdown patch. Enable workaround for exim mailbox format. Backport shingles static test. Fix levenshtein distance calculations. Fix max_train setup in ANN module. Fix redis structure by adding NULL, NULL member. Fix build with unmodified LibreSSL opensslv.h. Repair optional dependencies. Really skip filters in case of pre-result set. Restore the intended pre-filters behaviour. Fix ipv6 mask application.
1.2.629 Apr 2016 15:49
Fix parsing of URLs in texts. Fix creating of URLs from LUA. Fix some more URL detector issues. Fix unit tests. Fix JIT compilation for PCRE2 expressions. Fix JIT usage for PCRE2. Fix UTF8 mode in PCRE2. Add workaround for pre-historic compilers (#605). Fix and rescore R_PARTS_DIFFER logic. Properly set lua paths for tests. Fix SA rawbody processing - exclude top part. Store text parts content with newlines stripped. Properly support SA body regexps. Fix body rules in SA plugin. Fix setting of score for parts differ. More fixes to parts distance calculations. Fix percents output in R_PARTS_DIFFER. Plug memory leak in dkim module. Plug minor memory leak in regexps creation.
1.2.522 Apr 2016 20:52
Plug an important memory leak in headers getting code. Remove some bad domains from whitelists.
1.2.420 Apr 2016 14:41
Implement new multipattern matcher that uses hyperscan if possible. Use mutlipattern for lua_trie code. Add utility methods for multipattern. Use multipattern in url matcher. Add escape functions for hyperscan. Allow to optimize lua - C transition by flattening table args. Optimize hot paths in SA plugin. Optimize rspamd_re_cache_type_from_string. Allow empty tries. Fix extraction of URLs from Subject. Allow to have different flags for different patterns in multipattern. Add common directory for hyperscan cache to config. Implement caching for hyperscan multipattern. Attach domain part to `R_SUSPICIOUS_URL` (by @moisseev). Allow multipattern scans to be nested for the case of hyperscan. Simplify SURBL redirector search code and avoid ac_trie. Add two way substring search algorithm. Avoid acism usage to find gtube pattern. Fix processing of empty headers. Allow to disable pthread mutexes on broken platforms. Make web interface not send password in query strings (#585) by @fatalbanana. Add maximum delay to ratelimit module. Backport fix for empty files inclusion from libucl. Fix settings id setup. Add min_learns option to classifiers. Use more clever to utf8 conversion strategy. Fix disabling of virtual symbols in the settings. Rework settings to work properly in metric-less configuration. Set the default limit for classifier. Fix ttl based expiration from LRU cache. Rework DKIM module to use OpenSSL for digests. Fix mailto urls parsing with hyperscan. Do not set obscured flag for urls starting with spaces. Fix crash on redis learn. Fix ratelimit ctime setting.
1.2.312 Apr 2016 12:13
New DCC module (by @smfreegard). Rework whitelist module. Fix regexps results combination (*critical*). Fix issue with expressions processing (*critical*). Optimize strlcpy for aligned input. Add support of half-closed connection in lua_tcp. Allow to print compact json in client. Save required score in history (#581). Allow to attach file descriptors to control commands. Allow to send descriptors from workers to main. Allow to attach fd when broadcasting to workers. Implement log pipe feature for rspamd logs analysis. Add `log_helper` worker. Add `URIBL_SBL_CSS` (by @smfreegard). Add worker scripts functionality. Add on load hooks for rspamd_config. Add lua scripts for log_helper worker. Add generic maillist detector (#584). Implement FANN autolearn using log_helper worker. Rework metrics configuration to allow includes. Change default value of forced removal in composite rules. Allow to use assembly version of blake2b on x86 cpu. Use less precise (but faster) clock if possible. Insert redirected URL to the urls list. Allow to get and set callback data for rspamd symbols. Add binary heap implementation. Use binary heap for expire algorithms in the hash. Use `least frequent used` expiration strategy. Allow to get mime headers from a task. Add support for mime headers in `regexp` module. Update Exim patches (by @fatalbanana). Allow building rspamd with jemalloc. Save multipart boundaries. SA plugin changes. LUA API changes.
1.2.201 Apr 2016 14:23
Use HTTP Content-Type on non mime input if possible. Save log level when compressing log messages. Further rework of composite rules (add ' ' prefix). Add tracking for rspamd expressions. Store actions limits in metric result. Fix parsing of include/redirect with many records in SPF. Add method to disable symbols execution in the cache. Allow to disable checks from settings. Allow to select settings by id in HTTP query. Find URLs with ' r' and ' n' inside href attribute. Implement vectored mode for hyperscan (experimental). Improve client connection errors diagnostics. Allow to edit new files with signtool. Improve hashes performance on 32 bit platforms. Fix sorting of limits. Remove slow and unused rules `INVALID_EXIM_RECEIVED*`. Add expression:process_traced lua method. Allow tables in task:insert_result. Save trace for SA metas. Do not parse broken TLD parts in URLs. Investigate many border cases in URLs parser.
1.2.125 Mar 2016 16:18
Add list support to `mime types` module configuration (by @moisseev). Allow symbols params to be printed in logs. Fix `MIME_BAD_ATTACHMENT` false positives for MDN/DSN. Fix crashes on arm32. Do not classify message if some class is missing. Fix cryptobox cleanup. Remove multipart/report from bad mime types (#569). Improve logging for fuzzy hashes. Show map URLs in webui. Sort symbols in webui.
1.2.024 Mar 2016 02:05
New dynamic updates plugin.
Regular expressions map support.
Faster radix trie algorithm.
Faster siphash for AVX2 supporing CPUs (used in fuzzy hashes).
Allow quoted and slashed keys in map.
Add proper support of DNS resolvers balancing.
Rework includes and configuration system for better local changes support.
New keypairs framework for signing and encryption.
Added support for dynamic modules and workers.
Allow to dump configuration with help comments.
Rework once_received module.
- priority for `good_hosts`.
- If a good host has been found do not add once_received symbols.
- priorities for strict once_received.
- Add ability to whitelist IP addresses.
Implement support of signed maps for HTTP and file maps.
Add command to sync fuzzy storage.
Rework system of symbols and actions registration
It is possible now to use priorities when adding symbols to metrics and
override scores for symbols with lower priority with the scores with
Add auth support and db selection for redis stats.
Improve composite rules application.
Add ignore_received option.
critical with inconsistent resorting.
`all` in spf redirects.
Add punycoded versions for IDN domains.
Improve sorting order for symbols cache.
Add lockless logging for processes management.
Allow to specify flags for metric symbols.
Load images height and width from style attribute.
Override DNS requests limits for SPF and DKIM.
resetting symbols to their default values in WebUI.
Improve configuration agility for redis stats.
Allow to set db and password for redis in stat_convert.
Import the latest libucl.
LUA API changes:
- Add rspamd_version function to LUA API.
- Add lua_cryptobox module.
- Add lua_map module.
- Add task:set_metric_action lua API method.
- race condition in lua_tcp module.
- a lot of in lua_redis module.
- Rework and abstract lua maps API.
- Add util.strlen_utf8 lua function.
- Add lua functions for caseless comparison.
- Allow optional symbols registration.
- Add config:add_
1.1.422 Feb 2016 16:03
* Print traceback on lua errors in lua config
* Fix leaks in lua error paths
* Improve 'R_EMPTY_IMAGE' rule
* Fix metas memoization in SA plugin
* Properly set `flag` in fuzzy replies
* Fix arguments order
* Fix issue with out-of-boundary reading
* Fix issues found by coverity
* Same result checking error found by coverity
* Fix varargs processing (found by coverity)
* Fix error in printing hex
* Reduce weights for some hfilter patterns
* Add aliases
* Rework once_received module
* Fix `MISSING_MIMEOLE` rule for modern OE
* Treat meta tags as embedded tags (#501)
1.1.305 Feb 2016 03:15
DSN rules when SMTP from is unavailable.
statconvert routine to avoid lua module usage.
Set a sane quark for configtest to avoid NULL to be printed in logs.
Support c11 if available.
parsing of ip:port strings.
Add more diagnostic for lua subr errors.
task:set_from_ip lua method.
Add basic routines for digital signatures.
Add tool for digital signatures.
Add plain open file API method for atomic open.
parsing nested braces inside logger vars.
Pre filters now actually skip processing.
Add pre-filter mode for multimap.
Switch to apache 2 license.
1.1.229 Jan 2016 16:24
* Fix stat_cache closing
* Add checkpoints to sqlite3 learn cache
* Do not recompile lua generated headers all the time
* Increase number of messages learned
* Fix issues with dual stack and hfilter
* Disable MID checks for hfilter by default
* Fix cache definitions in multiple classifier and no type
* Don't crash if learn cache failed to initialize
* Fix googlegroups support in maillist plugin
* Rework flags LUA API
* Add `BROKEN_HEADERS` rule
* Add support for forged confirmation headers (by @AdUser)
* Allow `any`, `mime` and `smtp` for get_from/get_recipients
* Add mime types checking plugin
* Add rule to detect spammers attempts to cheat mime parsing
* Rework parsing of IP addresses in configuration (better IPv6 support)
* Add `util.parse_mail_address` function to LUA API
* Add lua sqlite3 module
* Implement synchronous redis call
* Ratelimit: avoid possible indexing of nil value (Fixes #498) (by @fatalbanana)
* Add stat_convert command to convert stats tokens from sqlite3 to redis
* Implement redis advanced lua api with pipelining
* Fix memory leak on redis stat (#500)
* Fix user/language learn count in sqlite statistics (#496) (by @fatalbanana)
* Fix build with custom pcre
* Fix fuzzy relearning (#498)
* Improve planning of asynchronous tasks
* Show slow rules in log
* Add warning for slow regexps
* Add base32 decode/encode routines to lua util
* Allow converting of learn cache from sqlite to redis
* Add methods to check if a messages has from/rcpts
* Improve and fix multimap plugin
* Disable reload command in rc scripts
* Improve runtime CPU dispatcher for libcryptobox
* Add preliminary support of digital signatures via ed25519
* Add detection for RDRAND support
* Print configuration of crypto on start
* A in SPF presumes AAAA lookup as well
1.1.121 Jan 2016 16:14
* Fix duplicated XBL symbol
* Reduce log severity for ratelimit missing servers
* Fix XBL composite to avoid duplicate symbols
* Reduce weight of URL_ONLY rule due to FP rate
* Disable fuzzy hashes from the metadata for now
* Fix processing of empty messages (#486)
* Always treat DNS timeouts as temporary fail for SPF
* Fix issue with SPF double IP stack (#483)
* Use X-Forwarded-For when checking secure_ip (#488)
* Fix hash calculation for sqlite stats
* Fix memory corruption on punycode
* Fix strings allocation in punycode
* Fix error message (#491)
1.1.018 Jan 2016 16:35
* Incompatible change: sqlite3 statistics and per_user behaviour: Now both redis and sqlite3 follow common principles for per-user
statistics: 1) If per-user statistics is enabled check per-user tokens ONLY 2) If per-user statistics is not enabled then check common tokens ONLY If you need old behaviour, then you'd need to use a separate classifier
for per-user statistics.
* Implement redis statistics backend and cache
* Implement autolearning for statistics
* Reworked statistics architecture from scratch
* Add hyperscan (https://github.com/01org/hyperscan) engine for regular expressions
* Implement flexible task logging
* Rework fuzzy worker
* Various fixes to SPF
* Add clang plugin for static analysis
* Add 'allow_raw_input' option for non-mime messages
* Recognize types using libmagic
* Fix parsing of IPv6 received headers
* Add new interface of communication between workers in rspamd
* Add support for named socketpairs
* Don't write URLs by default as it is too verbose
* Set status for HTTP replies
* Try load `rspamd.conf.override`
* Implement words decaying for text parts to limit many checks
* Improve support of SA rules and plugins
* Many fixes to the URL parser
* Match any newline character in regexps
* Fix resolving of upstreams and detection of poor IPv6 configurations
* Parse upstreams selection algorithm from the configuration line
* Add `reresolve` command to the control interface
* Generate fuzzy hashes from task metadata (URLs and headers)
* Add method to check if IP is local and `local_addrs` option
* Implement forced timeout for delayed filters
* Disable fast path of pcre-jit as it seems to be broken
* Bayes fixes
* Rules changes
* Plugins changes
* Rework ratelimit plugin
* Add neural network **experimental** plugin
* Add a sample script to learn neural network from rspamd logs
* Add documentation strings support to rspamd:
* Documentation improvements, tutorials section, statistics description
* Many other minor and major bugfixes not noted here
1.0.1121 Dec 2015 19:40
* Fix spf redirects
* Fix domains when parsing mx/ptr/a records in includes/redirects
* Fix unfolded base64 encoding
* Fix GError use-after-free
* Do not rewrite the original url when using redirector
* Fix parsing of fragment in urls
* Fix processing of HTML tags
* Improve empty image rule
* Avoid long double type
* Fix tokens weights in OSB algorithm
* Improve debugging for bayes
1.0.1006 Nov 2015 14:01
* Fix settings application (#416)
* Fix another issue with fixed strings
* Fix hash function invocation
* Use the proper string for make_dns_request in lua_http
* Fix scan time output
* Update webui:
- fix labels for greylisting
- fix dimension of scan time
1.0.928 Oct 2015 05:28
* Emergency fix in keyed blake2 to fix fuzzy hashes and encrypted password
* Support passwords longer than 64 symbols
* Add function to traverse AST atoms
* Allow dependencies on rspamd symbols for SA metas
* Fix memory corruption when timeout is removed in fuzzy check
* Fix encrypted fuzzy add processing
* Avoid use-after-free in controller session destructor
* Use session pool instead of task pool in fuzzy check
* Fix assembly in i386 mode (#413, #412)
1.0.726 Oct 2015 19:13
* Plugged memory leaks in internet address object html parser
* Fixed static build
* Fixed multiple sigchld processing
* Fixed deletion of signal events after event processing loop
* Fixed build on ARM (#404 - reported by @Gottox)
* Fixed setting the default mask for SPF.
* Fixed sanitisation of HTTP query values
* Fixed parsing of the last header in encrypted HTTP messages
* Additions and fixes for test suite benchmarks
* Added openssl aes-256-gcm support to libcryptobox HTTP server
* Implemented support for starting multiple HTTP servers
* Implemented batch accept in HTTP server
* Added module to get data from HTTP headers (#285 - reported by @msimerson)
* Added `rspamadm control` command
* Added ability to sort counters output.
* Added ability to specify custom headers for rspamc client
* Fix architecture detection
* Converted history storage to the UCL format
* Allow flexible number of rows in history
* Fix action badges in WebUI
* Add universal cryptobox hash API
* Migrated to the optimized blake2b implementation adopted from Andrew Moon
* Allow explicit loading of specific modules
* Always load settings module
* Allow to add symbols from settings
* Fix double free in the controller fuzzy learn command
* Avoid endless loop when cannot open sqlite db
* Updated libucl
1.0.616 Oct 2015 13:37
* Fix build on i386
* Update CentOS7 service file patch (by @fatalbanana)
* Fix path to rspamadm in Debian init script (by @fatalbanana)
* Fix broken '_SC_GETPW_R_SIZE_MAX' on FreeBSD
* Fix portability issues
* Use cryptobox chacha for libottery
* Better support of 32 bit builds
* Fix header name tokens setup
* Fix levenstein distance method for words
* Add workaround for old libevent (#400)
* Fix microseconds in termination timer
* Fix some more issues with fixed strings
* Explicitly test CPU instructions even after CPUID call
* Do not check out of boundary memory
* Do not output broken emails
* Fix unknown symbols registration
* Handle SIGILL using longjmp
* Block signals when exiting event loop
* Fix incorrect allocation size
* Slightly optimize alignment
* Restore rspamd -t for compatibility
* Add more sanity checks for emails
1.0.514 Oct 2015 17:24
* Add rspamd control interface:
- support `stat` command to get runtime stats of rspamd workers
- support `reload` command to reload runtime elements (e.g. sqlite3 databases)
* Rework curve25519 library for modular design:
- add Sandy2x implementation by Tung Chou
- fix CPU detection for variables loading assembly
- add testing for curve25519 ECDH
* New fixed strings library
* Add `R_SUSPICIOUS_IMAGES` rule
* Enable mmap in sqlite3
* Use new strings in the HTTP code
* Improve google perftools invocation
* Improve performance profiling in http test
* Reorganize includes to reduce namespace pollution
* Allow specific sections printing in configdump command
* Rework workers signals handlers to be chained if needed
* Update socketpair utility function
* Add control_path option for rspamd control protocol
* Fix ownership when listening on UNIX sockets
* Rework signals processing in main
* Remove extra tools from rspamd (they live in rspamadm now)
* Remove global rspamd_main
* Add global timeout for the overall task processing (8 seconds by default)
* Sanitize NULL values for fuzzy backend
* Store NM between encrypt/decrypt
* Add textpart:get_words_count method
* Fix generic DNS request in lua
* Tune hfilter weights
* Add support of IPv6 in hfilter
* Fix parsing of HTTP headers with IP addresses
* Sync with the recent libucl
* Various minor bugfixes
1.0.405 Oct 2015 14:13
* Add configdump routine to rspamadm
* Implement retransmits for fuzzy_check plugin
* Fix events processing for learning anf checking fuzzy hashes
* Avoid dependency on unneeded and uncompatible glib include
* Add `historyreset` command to the controller
* Fix loading of tokenizer config from dump (#389)
* Add sorting hints for the history
* Allow custom lua scripts for users/languages extraction (#388)
* Do not add FORGED_RECIPIENTS when 'To' is missing (#387)
* Do not add R_UNDISC_RCPT when 'To' is missing (#387)
* Add encryption to fuzzy check plugin
* Add encryption for fuzzy storage
* Add new epoch for encrypted fuzzy request
* Add encryption for `rspamd.com` storage
* Remove gmime processing for LDA mode as it is deadly broken
* Add routine to find end of headers position in mime messages
* Fix LDA headers folding
* Init libraries in rspamc client as well to avoid locale issues
* Avoid collision with locally installed includes
* Allocate and free memory with the same allocator in rspamadm (#385)
* Preserve expired fuzzy hashes counter
* Improvements in webui:
- Add favicon.ico
- Rework history table
- Fix sorting for the history
- Migrate to bootstrap 3 and jquery 2
- Fix css bugs
- Add glyphicons
- Add reset history
- Improve history buttons
- Redraw graph to avoid display issues
- Webui is now MIT licensed to match licensing policy of rspamd
1.0.328 Sep 2015 16:28
* Fix piechart clean slice (#380)
* Fix controller crashes when GString is reallocated (#381)
* Correctly set locale before start
* Set C locale for numeric values
* Add rspamadm routine:
- add `pw` command to manage passwords
- add `help` command for displaying help
- add `configtest` command to check configuration files
- add `keypair` command for generating encryption keys
- add `fuzzy_merge` routine to merge fuzzy sqlite databases
- add a simple manual page for rspamadm
* Allow metric registration for composite expressions
* Add strict mode for configtest
* Add logger counters
* Save and show learned messages count (#383)
* Add `no_stat` flag
* Add `task:set_flag` and `task:get_flags` (#382)
* Enable foreign keys in sqlite3
* Remove orphaned shingles from fuzzy storage
* Optimize synchronization steps for fuzzy storage
* Allow delayed conditions registration
* Add lua API for conditions registering
1.0.223 Sep 2015 17:01
* Fix critical bug in webui that prevents password from being sent
* Rework webui view:
* Add descriptions for whitelist maps
* Fix build on arm (#379)
* Fix issue with the last element in the radix trie
* Add more tests for radix trie algorithm
* Allow to extract URLs from query strings of other URLs (#361)
* Initialize rrd fields before writing to file
* Fix double free if no password has been specified
1.0.122 Sep 2015 16:08
* Add writing to rrd from the controller
* Fixed lots of bugs in rrd code
* Adopt new DNS API in hfilter plugin (by @AlexeySa)
* Allow only one controller process to manage rrd file
* Set event base for fuzzy calls
* Improve fuzzy IO errors logging
* Add rra extraction function to rrd library
* Add graph handler to the controller
* Cache correct passwords to avoid too high CPU usage when working with webui
* Controller sockets are owned by router do not export them to task
* Optimize logging by skipping hash table search if it's empty
* Fix loading issue whith broken statfiles
* Print assertions from glib to rspamd logger
* Load legacy `lua/rspamd.local.lua`
* Update webui with some fixes to learning and scanning
1.0.017 Sep 2015 15:51
* Reworked symbols processing
* Fixed URL redirector
* New sqlite3 backend
* Store tokenizer configuration within statfiles
* Improve bayes statistics:
* Rewritten url parsing
* Fix settings plugin to allow custom actions scores
* Improve rbl plugin
* Allow capturing patterns in rspamd lua regexp library
* Add GTUBE support
* Fix spamc legacy support
* Add DKIM support to RBL module
* Fix issues with multiple DKIM signatures
* Fix issue if rspamd cannot create statfiles (#331)
* Reworked parts and task structure
* Add ip_score plugin support (not enabled by default)
* Improve PARTS_DIFFER rule to count merely different words
* New HTML parser
* New version of LUA DNS API
* Table versions of many functions in LUA API
* Improved rspamc client
* Allow scanning of local files using HTTP requests
* Reworked configuration system:
* Rework and enable DMARC plugin
* Add whitelist plugin for SPF/DKIM/DMARC based whitelisting
* Add some common domains to whitelists shipped with rspamd
* Reworked logging
* Improved spamassassin plugin.
* Resurrect rrd support code
* Save controller stats between restarts
* Fixed tonns of bugs
* Added tonns of minor improvements and features
* Added more unit tests
* Create functional tests framework
* Added documentation for missing modules
* Added rpm/deb repositories and scripts
* Updated WebUI and libucl externals
0.9.1019 Aug 2015 19:02
* Do not dereference null pointer on learning.
* Fix some extreme cases in BAYES.
* Add a workaround to avoid bad HTML messages breaking.
* Build with -O2 flags by default.
* Add constraints to limit DNS requests count per task.
* Add workaround for SURBL DNS flood.
* Set error if rspamd cannot learn anything.
0.9.903 Jul 2015 14:07
* Don't use RWL_SPAMHAUS_WL (unknown result) for whitelisting (by @fatalbanana)
* Import updated public suffix list (by @fatalbanana)
* Remove debug message
* Fix settings (by @fatalbanana)
* Remove duplicated symbol registration
* Use WAL for fuzzy storage
* RBL fixes (by @fatalbanana): silence errors; yield unknown results from RBLs; fix scoring for DNSWL; fix use of RBL name as symbol; ignore RBL names that would not be yielded
* Support captures in regular expressions
* Add captures support to lua_regexp
* Support dist on FreeBSD and Darwin
* Add RCVD_IN_DNSWL_NONE as whitelisting exclusion (by @fatalbanana)
* Multiple fixes to URL detection: support port definition; fix query and path recognition; fix parsing of multiple slashes in URL; fix parsing query just after port; fix path field in `url:to_table` method; improve support of IP based URLs.
* Set ignore_whitelists = true for RECEIVED_SPAMHAUS_XBL (by @fatalbanana)
* Add GTUBE support
* Ignore User header in SA mode
0.9.825 Jun 2015 15:21
* Fix critical bug in bayes classifier (#305)
* Fix critical bug in RBL module (by @fatalbanana)
* Fix and rework settings plugin.
* Fix get_all_opts for a case of non-iterable options.
* Use tld for redirector's matching.
0.9.723 Jun 2015 16:18
* Add whitelist_exception setting to RBL module (by @fatalbanana)
* Don't use RWL_MAILSPIKE_POSSIBLE or DNSWL_BLOCKED for whitelisting (by @fatalbanana)
* Fix extreme cases in bayes classifier.
* Fix parsing of urls with '?' at the end of hostname.
* Update interface.
* Fix number of issues with webui interaction.
* Fix saving maps.
* Allow user@ and @domain matches in multimap.
* Fix issues with bounces From processing.
* Fix abs/fabs misuse.
* Fix builds on suse and arch linux distributions.
0.9.618 Jun 2015 12:37
* Fix memory leak if mime cannot be parsed.
* Fix dkim cache expiration.
* Fix issues with redirector HTTP response.
* Fix abnormal connection closing with certains messages with a high score (issue #296)
* Fix redirector installation.
* Use specific POE loop for some systems.
* Fix number of issues in URL redirector.
* Fix selecting URLs for sending to redirector.
0.9.529 May 2015 12:24
* Avoid double free when extending HTTP message.
* Fix double free if multiple classifiers are defined.
* Fix misprint in spamassassin plugin.
* Fix cpuid invocation on i386.
* Fix ownership issues for zero-copy decode.
* Allow __len metamethod on rspamd text .
* Add base64 decoding lua utility.
* Fix build on FreeBSD
* Skip spaces at the beginning of mime messages.
* DBL_ABUSE_REDIR should not have significant weight.
* Allow to split by lua_regexp rspamd text objects.
* Allow to specify custom stop pattern for lua_tcp.
0.9.421 May 2015 13:52
* Fix critical bugs in tokenization algorithm
* Write unit tests for tokenization
* Add documentation for lua_tcp
* Switch off legacy tokenization by default.
* Fix critical bugs in words normalization
* Add lua bindings to tokenizer.
* Implement storing of HTTP headers inside task
* Add lua API to accerss HTTP headers data
* Implemented base64 encoding suitable for MIME
* Use caseless hash and equal functions for HTTP request headers.
* Improve debian architectures support (by @dottedmag)
0.9.319 May 2015 16:25
* Revert incorrect regexp change that broke the default rules
* Fix lua_tcp module
0.9.219 May 2015 15:58
* Fix error on spawning unique workers.
* Add preliminary version of generic LUA TCP requests API.
* Use lua 5.1 if luajit is not available (Arm64, PowerPC, s390x etc)
* Fix fuzzy mime strings with only type.
* Improve thunderbird sanity checks.
* Fix critical bug on matching regular expressions.
* Make hiredis optional dependency.
* Fix multiple bugs in daemon reloading
0.9.117 May 2015 16:07
* Restore utf8 validation for regular expressions to avoid crashes
* Fix symbols displaying in the interface
* Add symbol groups to the interface
* Fix maps ID parsing in the controller
* Add multimap and regexp modules documentation
* Backport fixes from libucl
* Fix debian package (by @dottedmag)
* Rework XXH32 invocations
0.9.013 May 2015 21:11
* Add support of the fast and secure protocol level encryption:
* Rework expressions and create new expressions library:
* Many improvements to the LUA API:
* LuaJIT is now the default requirement for rspamd allowing to speed up lua execution by a large margin (however, plain lua is still supported)
* New plugins: SpamAssassin, DMARC
* New aho-corasic trie implementation from @mischasan that allows to load and use hundreds of thousands of patterns with no influence on load
* Support of PCRE JIT and PCRE JIT fast path modes that significantly improves the performance of regular expressions if supported by PCRE
* New URLs parser and extractor
* New statistics infrastructure
* Reworked utf8 tokenization that previously corrupted all UTF8 words (minor incompatibility with old fuzzy hashes with utf-8 symbols)
* SPF module has been completely rewritten to support complex cases of `include` and `redirect` within SPF records
* DKIM module now supports multiple signatures
* Controller passwords can now be stored encrypted by `PBKDF2-HMAC` in the configuration file
* Many hand-written HTTP clients has been replaced with the common rspamd http module
* New test framework
* Randomize hash seed to avoid certain hash tables vulnerabilities
* Documentation improvements:
* Fixed tonns of bugs and memory leaks
* Added tonns of minor features
0.8.315 Mar 2015 13:20
* Various critical fixes in distribution (by @dottedmag and @fatalbanana)
* Fixed bugs in url detector to parse certain patterns
* Add default host and helo for a client
* Some sanity checks for tokenizer and classifier
* Reiterate on systemd support
* Fix missing symbol registration
* Add support of spamc compatible output
* Filter double-dots in rbl.lua validate_dns (by @fatalbanana)
* Update ucl submodule due to critical bugfix