Redis module bloom filter 安装
redis加载RedisBloom整体流程
- 确认 RedisServer 版本。
- 根据 RedisServer 版本选取对应版本的 RedisBloom 。
- 修改 redis.conf 启用 bloom filter module 。
- 重启 RedisServer 加载 bloom filter module 。
(1) 确认RedisServer版本
[weikeqin@localhost redis (6.0 =) ]$ ./src/redis-server -v
Redis server v=6.0.20 sha=de0d9632:0 malloc=jemalloc-5.1.0 bits=64 build=db68ee83ba489356
[weikeqin@localhost redis (6.0 =) ]$
[weikeqin@localhost redis (6.0 =) ]$ pwd
/Users/weikeqin/WorkSpaces/middleware/redis
[weikeqin@localhost redis (6.0 =) ]$可以看到 redis-server 版本是 6.0.20
(2) 根据RedisServer选择对应的RedisBloom版本
RedisBloom各版本链接 https://github.com/RedisBloom/RedisBloom/releases
最新的 RedisBloom 2.6.3 要求 Redis版本 >= 7.2
由于使用的是Redis 6.0.20，能用到的 RedisBloom 版本是 2.4.0
RedisBloom 2.4.0版本链接 https://github.com/RedisBloom/RedisBloom/releases/tag/v2.4.0
(2.1) 下载对应版本的RedisBloom
[weikeqin@localhost middleware ]$ git clone git@github.com:RedisBloom/RedisBloom.git
Cloning into 'RedisBloom'...
remote: Enumerating objects: 5001, done.
remote: Counting objects: 100% (517/517), done.
remote: Compressing objects: 100% (173/173), done.
remote: Total 5001 (delta 397), reused 394 (delta 338), pack-reused 4484
Receiving objects: 100% (5001/5001), 1.62 MiB | 261.00 KiB/s, done.
Resolving deltas: 100% (3262/3262), done.
[weikeqin@localhost middleware ]$选择对应版本
[weikeqin@localhost middleware ]$ cd RedisBloom
[weikeqin@localhost RedisBloom (master =) ]$
[weikeqin@localhost RedisBloom (master =) ]$ git checkout -b 2.4 origin/2.4
Branch '2.4' set up to track remote branch '2.4' from 'origin'.
Switched to a new branch '2.4'
[weikeqin@localhost RedisBloom (2.4 =) ]$
[weikeqin@localhost RedisBloom (2.4 =) ]$ git lg2
* 7ab9e4a - (HEAD -> 2.4, origin/2.4) Update config.yml (5 months ago) <Guy Korland> (2023-04-03 09:57:52 +0300)
* b66ec56 - Update config.yml (5 months ago) <Guy Korland> (2023-04-03 09:55:52 +0300)
* b2b3fe1 - (tag: v2.4.5) Version 2.4.5 (#624) (6 months ago) <Rafi Einstein> (2023-02-27 16:17:59 +0200)
* 52e0373 - Merge pull request #623 from RedisBloom/rafi-2.4-amzn2-1 (6 months ago) <Rafi Einstein> (2023-02-25 12:12:26 +0200)
[weikeqin@localhost RedisBloom (2.4 =) ]$ (2.2) 使用 RedisBloom 2.4.0 源码编译可执行文件
如果使用 RedisBloom 2.4.0 源码编译可执行文件遇到的问题比较多，短时间解决不了，可以使用 RedisBloom 2.2.14，这个版本编译相对容易一些。
[weikeqin@localhost RedisBloom (2.4 =) ]$ git submodule init
Submodule 'deps/readies' (https://github.com/RedisLabsModules/readies.git) registered for path 'deps/readies'
Submodule 'deps/t-digest-c' (https://github.com/RedisBloom/t-digest-c) registered for path 'deps/t-digest-c'
[weikeqin@localhost RedisBloom (2.4 =) ]$
[weikeqin@localhost RedisBloom (2.4 =) ]$
[weikeqin@localhost RedisBloom (2.4 =) ]$ git submodule update
Cloning into '/Users/weikeqin/WorkSpaces/middleware/RedisBloom/deps/readies'...
Cloning into '/Users/weikeqin/WorkSpaces/middleware/RedisBloom/deps/t-digest-c'...
Submodule path 'deps/readies': checked out '59712b120ed6d8b39fde438d3860ca8e7262e6c1'
Submodule path 'deps/t-digest-c': checked out '9dcd73d876308a76e10dd0fc5a0a669b264898aa'
[weikeqin@localhost RedisBloom (2.4 =) ]$ 编译
[weikeqin@localhost RedisBloom (2.4 =) ]$ make
Building /Users/weikeqin/WorkSpaces/middleware/RedisBloom/bin/macos-x64-release/t-digest-c/src/libtdigest_static.a ...
Generating /Users/weikeqin/WorkSpaces/middleware/RedisBloom/bin/macos-x64-release/t-digest-c/Makefile ...
...
-- Generating done
-- Build files have been written to: /Users/weikeqin/WorkSpaces/middleware/RedisBloom/bin/macos-x64-release/t-digest-c
Building /Users/weikeqin/WorkSpaces/middleware/RedisBloom/bin/macos-x64-release/t-digest-c/libtdigest_static.a ...
[ 50%] Building C object src/CMakeFiles/tdigest_static.dir/tdigest.c.o
[100%] Linking C static library libtdigest_static.a
[100%] Built target tdigest_static
Compiling deps/bloom/bloom.c...
...
Linking /Users/weikeqin/WorkSpaces/middleware/RedisBloom/bin/macos-x64-release/redisbloom.so...
[weikeqin@localhost RedisBloom (2.4 =) ]$ RedisBloom 2.4 编译后的 redisbloom.so 文件在 RedisBloom/bin/macos-x64-release/redisbloom.so
(2.3) 使用 RedisBloom 2.2.14 源码编译可执行文件
[weikeqin@localhost RedisBloom (master =) ]$ git tag
v1.0.0
v1.0.1
v1.0.2
v1.0.3
v1.1.0
v1.1.1
v1.99.0
v1.99.1
v1.99.2
v2.0.0
v2.0.1
v2.0.2
v2.0.3
v2.2.0
v2.2.1
v2.2.10
v2.2.11
v2.2.12
v2.2.13
v2.2.14
v2.2.15
v2.2.16
v2.2.17
v2.2.18
v2.2.2
v2.2.3
v2.2.4
v2.2.5
v2.2.6
v2.2.7
v2.2.8
v2.2.9
v2.4.0
v2.4.1
v2.4.2
v2.4.3
v2.4.4
v2.4.5
v2.6.0
v2.6.1
v2.6.2
v2.6.3
ver2.2.15
[weikeqin@localhost RedisBloom (master =) ]$选择对应的的tag v2.2.14
[weikeqin@localhost RedisBloom (2.4 =) ]$ git checkout -b v2.2.14 v2.2.14
warning: unable to rmdir 'deps/t-digest-c': Directory not empty
M deps/readies
Switched to a new branch 'v2.2.14'
[weikeqin@localhost RedisBloom (v2.2.14) ]$
[weikeqin@localhost RedisBloom (v2.2.14) ]$ git status
On branch v2.2.14编译
[weikeqin@localhost RedisBloom (v2.2.14) ]$ make
clang -Wall -Wno-unused-function -g -ggdb -O2 -fPIC -std=gnu99 -D_GNU_SOURCE -fcommon -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom/contrib -c -o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/rebloom.o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/rebloom.c
clang -Wall -Wno-unused-function -g -ggdb -O2 -fPIC -std=gnu99 -D_GNU_SOURCE -fcommon -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom/contrib -c -o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/contrib/MurmurHash2.o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/contrib/MurmurHash2.c
clang -Wall -Wno-unused-function -g -ggdb -O2 -fPIC -std=gnu99 -D_GNU_SOURCE -fcommon -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom/contrib -c -o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/rmutil/util.o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/rmutil/util.c
clang -Wall -Wno-unused-function -g -ggdb -O2 -fPIC -std=gnu99 -D_GNU_SOURCE -fcommon -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom/contrib -c -o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/sb.o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/sb.c
clang -Wall -Wno-unused-function -g -ggdb -O2 -fPIC -std=gnu99 -D_GNU_SOURCE -fcommon -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom/contrib -c -o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/cf.o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/cf.c
clang -Wall -Wno-unused-function -g -ggdb -O2 -fPIC -std=gnu99 -D_GNU_SOURCE -fcommon -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom/contrib -c -o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/rm_topk.o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/rm_topk.c
clang -Wall -Wno-unused-function -g -ggdb -O2 -fPIC -std=gnu99 -D_GNU_SOURCE -fcommon -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom/contrib -c -o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/topk.o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/topk.c
clang -Wall -Wno-unused-function -g -ggdb -O2 -fPIC -std=gnu99 -D_GNU_SOURCE -fcommon -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom/contrib -c -o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/rm_cms.o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/rm_cms.c
clang -Wall -Wno-unused-function -g -ggdb -O2 -fPIC -std=gnu99 -D_GNU_SOURCE -fcommon -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom -I/Users/weikeqin/WorkSpaces/middleware/RedisBloom/contrib -c -o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/cms.o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/cms.c
ld /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/rebloom.o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/contrib/MurmurHash2.o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/rmutil/util.o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/sb.o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/cf.o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/rm_topk.o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/topk.o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/rm_cms.o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/src/cms.o -o /Users/weikeqin/WorkSpaces/middleware/RedisBloom/redisbloom.so -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -dylib -exported_symbol _RedisModule_OnLoad -lm -lc
[weikeqin@localhost RedisBloom (v2.2.14) ]$[weikeqin@localhost RedisBloom (v2.2.14) ]$ pwd
/Users/weikeqin/WorkSpaces/middleware/RedisBloom
[weikeqin@localhost RedisBloom (v2.2.14) ]$
[weikeqin@localhost RedisBloom (v2.2.14) ]$ ll
total 392
drwxr-xr-x 29 weikeqin staff 928 Oct 15 14:35 ./
drwxr-xr-x 17 weikeqin staff 544 Aug 26 18:59 ../
drwxr-xr-x 3 weikeqin staff 96 Oct 15 14:34 .circleci/
-rw-r--r-- 1 weikeqin staff 115 Aug 26 18:53 .clang-format
drwxr-xr-x 13 weikeqin staff 416 Oct 15 14:35 .git/
drwxr-xr-x 3 weikeqin staff 96 Oct 15 14:34 .github/
-rw-r--r-- 1 weikeqin staff 157 Oct 15 14:34 .gitignore
-rw-r--r-- 1 weikeqin staff 200 Oct 15 14:34 .gitmodules
-rw-r--r-- 1 weikeqin staff 431 Oct 15 14:34 Dockerfile
-rw-r--r-- 1 weikeqin staff 5789 Oct 15 14:34 LICENSE
-rw-r--r-- 1 weikeqin staff 3948 Oct 15 14:34 Makefile
-rw-r--r-- 1 weikeqin staff 4640 Oct 15 14:34 README.md
drwxr-xr-x 3 weikeqin staff 96 Aug 26 19:08 bin/
drwxr-xr-x 3 weikeqin staff 96 Oct 15 14:34 build/
-rw-r--r-- 1 weikeqin staff 211 Aug 26 18:53 changelog
-rw-r--r-- 1 weikeqin staff 520 Aug 26 18:53 codecov.yml
drwxr-xr-x 7 weikeqin staff 224 Oct 15 14:35 contrib/
drwxr-xr-x 4 weikeqin staff 128 Oct 15 14:34 deps/
drwxr-xr-x 13 weikeqin staff 416 Oct 15 14:34 docs/
-rwxr-xr-x 1 weikeqin staff 715 Oct 15 14:34 getver*
-rw-r--r-- 1 weikeqin staff 1464 Oct 15 14:34 mkdocs.yml
drwxr-xr-x 3 weikeqin staff 96 Oct 15 14:34 opt/
-rwxr-xr-x 1 weikeqin staff 5767 Oct 15 14:34 pack.sh*
-rw-r--r-- 1 weikeqin staff 570 Oct 15 14:34 ramp.yml
-rwxr-xr-x 1 weikeqin staff 128064 Oct 15 14:35 redisbloom.so*
drwxr-xr-x 5 weikeqin staff 160 Oct 15 14:35 rmutil/
drwxr-xr-x 30 weikeqin staff 960 Oct 15 14:35 src/
-rwxr-xr-x 1 weikeqin staff 1756 Oct 15 14:34 system-setup.py*
drwxr-xr-x 20 weikeqin staff 640 Oct 15 14:34 tests/
[weikeqin@localhost RedisBloom (v2.2.14) ]$ RedisBloom 2.2.14 编译后的 redisbloom.so 在 RedisBloom/src/rebloom.o
(3) 修改RedisServer配置redis.conf启用bloomfiler
################################## MODULES #####################################
# Load modules at startup. If the server is not able to load modules
# it will abort. It is possible to use multiple loadmodule directives.
#
# loadmodule /path/to/my_module.so
# loadmodule /path/to/other_module.so
loadmodule /Users/weikeqin/WorkSpaces/middleware/RedisBloom/bin/macos-x64-release/redisbloom.so(4) 启动RedisServer
(4.1) Debug模式启动RedisServer
Warning: Debuggee TargetArchitecture not detected, assuming x86_64.
=thread-selected,id="1"
=library-unloaded,id="/Users/weikeqin/WorkSpaces/middleware/redis/src/redis-server",target-name="/Users/weikeqin/WorkSpaces/middleware/redis/src/redis-server",host-name="/Users/weikeqin/WorkSpaces/middleware/redis/src/redis-server"
Loaded '/usr/lib/dyld'. Symbols loaded.
Loaded '/Users/weikeqin/WorkSpaces/middleware/redis/src/redis-server'. Symbols loaded.
Loaded '/usr/lib/libSystem.B.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libcache.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libcommonCrypto.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libcompiler_rt.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libcopyfile.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libcorecrypto.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libdispatch.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libdyld.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libkeymgr.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libmacho.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libquarantine.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libremovefile.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_asl.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_blocks.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_c.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_collections.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_configuration.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_containermanager.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_coreservices.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_darwin.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_dnssd.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_featureflags.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_info.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_m.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_malloc.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_networkextension.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_notify.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_product_info_filter.dylib'. Cannot find or open the symbol file.
Loaded '/usr/lib/system/libsystem_sandbox.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_secinit.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_kernel.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_platform.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_pthread.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_symptoms.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_trace.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libunwind.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libxpc.dylib'. Symbols loaded.
Loaded '/usr/lib/libc++abi.dylib'. Symbols loaded.
Loaded '/usr/lib/libobjc.A.dylib'. Symbols loaded.
Loaded '/usr/lib/liboah.dylib'. Symbols loaded.
Loaded '/usr/lib/libc++.1.dylib'. Symbols loaded.
Execute debugger commands using "-exec <command>", for example "-exec info registers" will list registers in use (when GDB is the debugger)
9601:C 26 Aug 2023 19:38:56.924 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
9601:C 26 Aug 2023 19:38:56.924 # Redis version=6.0.20, bits=64, commit=de0d9632, modified=0, pid=9601, just started
9601:C 26 Aug 2023 19:38:56.924 # Configuration loaded
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 6.0.20 (de0d9632/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 9601
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
9601:M 26 Aug 2023 19:38:57.776 # Server initialized
Loaded '/Users/weikeqin/WorkSpaces/middleware/RedisBloom/bin/macos-x64-release/redisbloom.so'. Symbols loaded.
9601:M 26 Aug 2023 19:38:57.815 * <bf> RedisBloom version 2.4.5 (Git=unknown)
9601:M 26 Aug 2023 19:38:57.817 * Module 'bf' loaded from /Users/weikeqin/WorkSpaces/middleware/RedisBloom/bin/macos-x64-release/redisbloom.so
9601:M 26 Aug 2023 19:38:57.819 * Ready to accept connections
The program '/Users/weikeqin/WorkSpaces/middleware/redis/src/redis-server' has exited with code 0 (0x00000000).(4.2) 正常模式启动
[weikeqin@localhost redis (6.0 =) ]$
[weikeqin@localhost redis (6.0 =) ]$ ./src/redis-server redis.conf
9838:C 26 Aug 2023 19:40:07.623 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
9838:C 26 Aug 2023 19:40:07.623 # Redis version=6.0.20, bits=64, commit=de0d9632, modified=0, pid=9838, just started
9838:C 26 Aug 2023 19:40:07.624 # Configuration loaded
9838:M 26 Aug 2023 19:40:07.626 * Increased maximum number of open files to 10032 (it was originally set to 256).
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 6.0.20 (de0d9632/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 9838
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
9838:M 26 Aug 2023 19:40:07.629 # Server initialized
9838:M 26 Aug 2023 19:40:07.630 * <bf> RedisBloom version 2.4.5 (Git=unknown)
9838:M 26 Aug 2023 19:40:07.630 * Module 'bf' loaded from /Users/weikeqin/WorkSpaces/middleware/RedisBloom/bin/macos-x64-release/redisbloom.so
9838:M 26 Aug 2023 19:40:07.630 * Ready to accept connections
^C9838:signal-handler (1693050020) Received SIGINT scheduling shutdown...
9838:M 26 Aug 2023 19:40:20.879 # User requested shutdown...
9838:M 26 Aug 2023 19:40:20.879 * Saving the final RDB snapshot before exiting.
9838:M 26 Aug 2023 19:40:20.880 * DB saved on disk
9838:M 26 Aug 2023 19:40:20.880 * Removing the pid file.
9838:M 26 Aug 2023 19:40:20.880 # Redis is now ready to exit, bye bye...
[weikeqin@localhost redis (6.0 =) ]$ 至此，RedisServer 加载 redisbloom.so 完成。
可以在Redis里使用 bloom filter 了。
(5) 遇到的问题
(5.1) ModuleNotFoundError: No module named ‘paella’
[weikeqin@localhost RedisBloom-2.4.3 ]$ ./sbin/setup
./sbin/setup: line 7: /Users/weikeqin/Software/redis/RedisBloom-2.4.3/deps/readies/shibumi/defs: No such file or directory
./sbin/setup: line 9: /Users/weikeqin/Software/redis/RedisBloom-2.4.3/deps/readies/bin/getpy3: No such file or directory
Traceback (most recent call last):
File "/Users/weikeqin/Software/redis/RedisBloom-2.4.3/sbin/system-setup.py", line 11, in <module>
import paella
ModuleNotFoundError: No module named 'paella'
[weikeqin@localhost RedisBloom-2.4.3 ]$ RedisBloom-2.4.3 的安装脚本里用到了 python3 的 paella 库
对应源码在
安装一下就好了
[weikeqin@localhost RedisBloom-2.4.3 ]$ pip3 install paella
Collecting paella
Downloading paella-0.0.1.dev1-py3-none-any.whl (26 kB)
...
Using legacy 'setup.py install' for warmup-scheduler, since package 'wheel' is not installed.
Using legacy 'setup.py install' for pathtools, since package 'wheel' is not installed.
Installing collected packages: wcwidth, warmup-scheduler, tokenizers, sentencepiece, safetensors, pathtools, mpmath, appdirs, urllib3, tqdm, sympy, smmap, six, setproctitle, regex, pyyaml, pyparsing, psutil, protobuf, Pillow, numpy, networkx, MarkupSafe, kiwisolver, idna, ftfy, fsspec, fonttools, filelock, cycler, Click, charset-normalizer, certifi, sentry-sdk, requests, python-dateutil, jinja2, gitdb, docker-pycreds, contourpy, torch, matplotlib, huggingface-hub, GitPython, wandb, transformers, torchvision, timm, open-clip-torch, paella
Running setup.py install for warmup-scheduler ... done
Running setup.py install for pathtools ... done
Successfully installed Click-8.1.7 GitPython-3.1.32 MarkupSafe-2.1.3 Pillow-10.0.0 appdirs-1.4.4 certifi-2023.7.22 charset-normalizer-3.2.0 contourpy-1.1.0 cycler-0.11.0 docker-pycreds-0.4.0 filelock-3.12.2 fonttools-4.42.1 fsspec-2023.6.0 ftfy-6.1.1 gitdb-4.0.10 huggingface-hub-0.16.4 idna-3.4 jinja2-3.1.2 kiwisolver-1.4.5 matplotlib-3.7.2 mpmath-1.3.0 networkx-3.1 numpy-1.25.2 open-clip-torch-2.20.0 paella-0.0.1.dev1 pathtools-0.1.2 protobuf-3.20.3 psutil-5.9.5 pyparsing-3.0.9 python-dateutil-2.8.2 pyyaml-6.0.1 regex-2023.8.8 requests-2.31.0 safetensors-0.3.3 sentencepiece-0.1.99 sentry-sdk-1.29.2 setproctitle-1.3.2 six-1.16.0 smmap-5.0.0 sympy-1.12 timm-0.9.5 tokenizers-0.13.3 torch-2.0.1 torchvision-0.15.2 tqdm-4.66.1 transformers-4.32.0 urllib3-2.0.4 wandb-0.15.8 warmup-scheduler-0.3 wcwidth-0.2.6
[weikeqin@localhost RedisBloom-2.4.3 ]$(5.2) ‘tdigest.h’ file not found
[weikeqin@localhost RedisBloom (2.4 =) ]$ make
Building /Users/weikeqin/WorkSpaces/middleware/RedisBloom/bin/macos-x64-release/t-digest-c/src/libtdigest_static.a ...
Generating /Users/weikeqin/WorkSpaces/middleware/RedisBloom/bin/macos-x64-release/t-digest-c/Makefile ...
/bin/bash: cmake: command not found
make[1]: *** [/Users/weikeqin/WorkSpaces/middleware/RedisBloom/deps/readies/mk/cmake.rules:17: /Users/weikeqin/WorkSpaces/middleware/RedisBloom/bin/macos-x64-release/t-digest-c/Makefile] Error 127
make: *** [Makefile:191: /Users/weikeqin/WorkSpaces/middleware/RedisBloom/bin/macos-x64-release/t-digest-c/src/libtdigest_static.a] Error 2
[weikeqin@localhost RedisBloom (2.4 =) ]$参考 https://github.com/RedisBloom/RedisBloom/issues/296
(5.3) Make version is too old
[weikeqin@localhost RedisBloom (2.4 =) ]$ make
deps/readies/mk/main:6: *** GNU Make version is too old. Aborting.. Stop.
[weikeqin@localhost RedisBloom (2.4 =) ]$ [weikeqin@localhost ~ ]$ brew install make
==> Downloading https://mirrors.aliyun.com/homebrew/homebrew-bottles/make-4.3.monterey.bottle.1.tar.gz
######################################################################## 100.0%
==> Pouring make-4.3.monterey.bottle.1.tar.gz
==> Caveats
GNU "make" has been installed as "gmake".
If you need to use it as "make", you can add a "gnubin" directory
to your PATH from your bashrc like:
PATH="/usr/local/opt/make/libexec/gnubin:$PATH"
==> Summary
🍺 /usr/local/Cellar/make/4.3: 15 files, 1MB
==> Running `brew cleanup make`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
[weikeqin@localhost ~ ]$
[weikeqin@localhost ~ ]$ make -v
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for i386-apple-darwin11.3.0
[weikeqin@localhost ~ ]$[weikeqin@localhost ~ ]$ /usr/local/opt/make/libexec/gnubin/make -v
GNU Make 4.3
Built for x86_64-apple-darwin21.1.0
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
[weikeqin@localhost ~ ]$# config
PATH="/usr/local/opt/make/libexec/gnubin:$PATH"[weikeqin@localhost RedisBloom (2.4 =) ]$ make -v
GNU Make 4.3
Built for x86_64-apple-darwin21.1.0
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
[weikeqin@localhost RedisBloom (2.4 =) ]$参考 mac 提示 GNU Make version is too old
参考资料
[1] bloom-filter介绍-redis.io
[2] Probabilistic data structures
[3] Probably and No: Redis, Probabilistic, and Bloom Filters
[4] Bloom Filter Datatype for Redis
[5] Low latency and compact probabilistic data structures