本文共 17428 字，大约阅读时间需要 58 分钟。
1.技术背景介绍;
FastDFS是一个开源的轻量级分布式文件系统，它对文件进行管理，功能包括：文件存储、文件同步、文件访问（文件上传、文件下载）等，解决了大容量存储和负载均衡的问题。特别适合以文件为载体的在线服务，如相册网站、视频网站等等。
FastDFS为互联网量身定制，充分考虑了冗余备份、负载均衡、线性扩容等机制，并注重高可用、高性能等指标，使用FastDFS很容易搭建一套高性能的文件服务器集群提供文件上传、下载等服务。2.基础环境介绍;
主机名 IP地址规划 相关软件源码包汇总 服务器角色
Nginx_fastdfs1
192.168.1.190 Nginx.1.8.0.tar.gzfastdfs .v5.05.tar.gzlibfastcommon-master.zipfastdfs-nginx-module_v1.16.tar.gz Storage/trackerNginx_fastdfs2
192.168.1.200 Storage/tracker3.软件功能介绍;
(1) libfastcommon 源码包介绍；libfastcommon是从开源软件FastDFS和FastDHT中提取出来的公共C函数库。
它提供了字符串处理、链表、哈希表、日志记录、配置文件读取、socket操作、base64编码、url编码、md5等常用函数库。还包括http的一个简单的client，可以获取url内容（目前仅支持GET方式）。这些代码已经在FastDFS和FastDHT中使用，比较简洁、稳定。FastDFS已经在Linux、FreeBSD、AIX下进行了编译并通过了测试.(2)fastdfs-nginx-module_v1 介绍;
我们在使用FastDFS部署一个分布式文件系统的时候，通过FastDFS的客户端API来进行文件的上传、下载、删除等操作。同时通过FastDFS 的HTTP服务器来提供HTTP服务。但是FastDFS的HTTP服务较为简单，无法提供负载均衡等高性能的服务，所以FastDFS的开发者——淘宝 的架构师余庆同学，为我们提供了Nginx上使用的FastDFS模块。其使用非常简单。(3) fastdfs 介绍;
FastDFS是一个开源的轻量级分布式文件系统，它对文件进行管理，功能包括：文件存储、文件同步、文件访问（文件上传、文件下载）等，解决了大容量存储和负载均衡的问题。特别适合以文件为载体的在线服务，如相册网站、视频网站等等。stDFS服务端有两个角色：跟踪器（tracker）和存储节点（storage）。跟踪器主要做调度工作，在访问上起负载均衡的作用。存储节点存储文件，完成文件管理的所有功能：存储、同步和提供存取接 口，FastDFS同时对文件的metadata进行管理。所谓文件的meta data就是文件的相关属性，以键值对（key valuepair）方式表示，如：width=1024，其中的key为width，value为1024。文件metadata是文件属性列表，可以包含多个键值对。跟踪器和存储节点都可以由一台或多台服务器构成。跟踪器和存储节点中的服务器均可以随时增加或下线而不会影响线上服务。其中跟踪器中的所有服务器都是对等的，可以根据服务器的压力情况随时增加或减少。为了支持大容量，存储节点（服务器）采用了分卷（或分组）的组织方式。存储系统由一个或多个卷组成，卷与卷之间的文件是相互独立的，所有卷的文件容量累加就是整个存储系统中的文件容量。一个卷可以由一台或多台存储服务器组成，一个卷下的存储服务器中的文件都是相同的，卷中的多台存储服务器起到了冗余备份和负载均衡的作用。在卷中增加服务器时，同步已有的文件由系统自动完成，同步完成后，系统自动将新增服务器切换到线上提供服务。当存储空间不足或即将耗尽时，可以动态添加卷。只需要增加一台或多台服务器，并将它们配置为一个新的卷，这样就扩大了存储系统的容量。FastDFS中的文件标识分为两个部分：卷名和文件名，二者缺一不可。(4).上传交互过程;
(5).下载交互过程
3.部署环境准备;
注意:(两台机器同样执行以下操作）
(1).安装环境依赖包
yum install -y zlib zlib-devel pcre pcre-devel gcc gcc-c++ openssl openssl-devel libevent libevent-devel perl unzip
(2).创建fastdfs 运行用户;
useradd -s /sbin/nologin fastdfs
(3).创建数据存储目录；
mkdir -p /export/fastdfs/{storage,tracker}
[root@nginx_Fastdfs1 ~]# ll /export/fastdfs/total 8drwxr-xr-x 4 fastdfs fastdfs 4096 Jun 6 22:09 storagedrwxr-xr-x 4 fastdfs fastdfs 4096 Jun 6 21:12 tracker[root@nginx_Fastdfs2 ~]# ll /export/fastdfs/
total 8drwxr-xr-x 4 fastdfs fastdfs 4096 Jun 6 21:59 storagedrwxr-xr-x 4 fastdfs fastdfs 4096 Jun 6 21:03 tracker4.Fastdfs部署详情;
注意：（2台服务器需要同样执行如下操作）
(1).部署libfastcommon;
mkdir /usr/local/fastdfs_packagescd /usr/local/fastdfs_packageswget unzip master.zipcd libfastcommon-master/./make.sh./make.sh install(2)部署fastdfs
cd /usr/local/fastdfs_packageswget %20Code/FastDFS%20Server%20with%20PHP%20Extension%20Source%20Code%20V5.05/FastDFS_v5.05.tar.gztar zxf FastDFS_v5.05.tar.gz && cd FastDFS./make.sh && ./make.sh install\cp pa conf/.conf /etc/fdfs/cd /etc/fdfs/mv .sample /opt/ #备份模板文件;chown -R fastdfs: /export/fastdfs注意:配置tracker 和 storage
Nginx_fastdfs1 和Nginx_fastdfs2分别配置;(3) Nginx_fastdfs1机器（配置tracker 和 storage 配置文件;）
[root@nginx_Fastdfs1 fdfs]# tree /etc/fdfs/
/etc/fdfs/├── client.conf├── mod_fastdfs.conf├── storage.conf└── tracker.conf0 directories, 4 filesstorage 配置
[root@nginx_Fastdfs1 /]# cat /etc/fdfs/storage.conf
group_name=group1base_path=/export/fastdfs/storagestore_path0=/export/fastdfs/storagetracker_server=192.168.1.190:22122tracker_server=192.168.1.200:22122http.server_port=80disabled=falsebind_addr=192.168.1.190client_bind=trueport=23000connect_timeout=30network_timeout=60heart_beat_interval=30stat_report_interval=60max_connections=256buff_size = 256KBwork_threads=2disk_rw_separated = truedisk_rw_direct = falsedisk_reader_threads = 1disk_writer_threads = 1sync_wait_msec=50sync_interval=0sync_start_time=00:00sync_end_time=23:59write_mark_file_freq=500store_path_count=1subdir_count_per_path=256log_level=inforun_by_group=run_by_user=allow_hosts=*file_distribute_path_mode=0file_distribute_rotate_count=100fsync_after_written_bytes=0sync_log_buff_interval=10sync_binlog_buff_interval=10sync_stat_file_interval=300thread_stack_size=512KBupload_priority=10if_alias_prefix=check_file_duplicate=0key_namespace=FastDFSkeep_alive=0http.disabled=falsehttphttp.domain_name=http.trunk_size=256KBhttp.need_find_content_type=truetracker.conf 配置;
[root@nginx_Fastdfs1 fdfs]# cat /etc/fdfs/tracker.conf
base_path=/export/fastdfs/trackerdisabled=falsebind_addr=192.168.1.190port=22122connect_timeout=30network_timeout=60max_connections=256work_threads=2store_lookup=2store_group=group2store_server=0store_path=0download_server=0reserved_storage_space = 4GBlog_level=inforun_by_group=run_by_user=allow_hosts=*sync_log_buff_interval = 10check_active_interval = 120thread_stack_size = 64KBstorage_ip_changed_auto_adjust = truestorage_sync_file_max_delay = 86400storage_sync_file_max_time = 300use_trunk_file = falseslot_min_size = 256slot_max_size = 16MBtrunk_file_size = 64MBhttp.disabled=falsehttp.server_port=8080http.check_alive_interval=30http.check_alive_type=tcphttp.check_alive_uri=/status.htmlhttp.need_find_content_type=truemod_fastdfs.conf 配置
[root@nginx_Fastdfs1 /]# cat /etc/fdfs/mod_fastdfs.conf
group_name=group1base_path=/export/fastdfs/storagestore_path0=/export/fastdfs/storagetracker_server=192.168.1.190:22122tracker_server=192.168.1.200:22122url_have_group_name = trueclient.conf 配置
[root@nginx_Fastdfs1 fdfs]# cat client.conf
tracker_server=192.168.1.190:22122tracker_server=192.168.1.200:22122(3) Nginx_fastdfs2机器（配置tracker 和 storage 配置文件;）
[root@nginx_Fastdfs2 ~]# tree /etc/fdfs/
/etc/fdfs/├── client.conf├── mod_fastdfs.conf├── storage.conf└── tracker.conf0 directories, 4 filesstorage.conf 配置；
[root@nginx_Fastdfs2 ~]# cat /etc/fdfs/storage.conf
base_path=/export/fastdfs/storagestore_path0=/export/fastdfs/storagetracker_server=192.168.1.200:22122tracker_server=192.168.1.190:22122http.server_port=80disabled=falsegroup_name=group1bind_addr=192.168.1.200client_bind=trueport=23000connect_timeout=30network_timeout=60heart_beat_interval=30stat_report_interval=60max_connections=256buff_size = 256KBwork_threads=2disk_rw_separated = truedisk_rw_direct = falsedisk_reader_threads = 1disk_writer_threads = 1sync_wait_msec=50sync_interval=0sync_start_time=00:00sync_end_time=23:59write_mark_file_freq=500store_path_count=1subdir_count_per_path=256log_level=inforun_by_group=run_by_user=allow_hosts=*file_distribute_path_mode=0file_distribute_rotate_count=100fsync_after_written_bytes=0sync_log_buff_interval=10sync_binlog_buff_interval=10sync_stat_file_interval=300thread_stack_size=512KBupload_priority=10if_alias_prefix=check_file_duplicate=0key_namespace=FastDFSkeep_alive=0http.disabled=falsehttphttp.domain_name=http.trunk_size=256KBhttp.need_find_content_type=truetracker.conf 配置
[root@nginx_Fastdfs2 ~]# cat /etc/fdfs/tracker.conf
base_path=/export/fastdfs/trackerdisabled=falsebind_addr=192.168.1.200port=22122connect_timeout=30network_timeout=60max_connections=256work_threads=2store_lookup=2store_group=group2store_server=0store_path=0download_server=0reserved_storage_space = 4GBlog_level=inforun_by_group=run_by_user=allow_hosts=*sync_log_buff_interval = 10check_active_interval = 120thread_stack_size = 64KBstorage_ip_changed_auto_adjust = truestorage_sync_file_max_delay = 86400storage_sync_file_max_time = 300use_trunk_file = falseslot_min_size = 256slot_max_size = 16MBtrunk_file_size = 64MBhttp.disabled=falsehttp.server_port=8080http.check_alive_interval=30http.check_alive_type=tcphttp.check_alive_uri=/status.htmlhttp.need_find_content_type=truemod_fastdfs.conf 配置
[root@nginx_Fastdfs2 ~]# cat /etc/fdfs/mod_fastdfs.conf
group_name=group1base_path=/export/fastdfs/storagestore_path0=/export/fastdfs/storagetracker_server=192.168.1.200:22122tracker_server=192.168.1.190:22122url_have_group_name = trueclient.conf 配置[root@nginx_Fastdfs2 ~]# cat /etc/fdfs/client.conf
tracker_server=192.168.1.200:22122tracker_server=192.168.1.190:221225.Fastdfs结合nginx;
(1).软件部署以及配置调整；
cd /usr/local/fastdfs_packageswget tar zxf fastdfs-nginx-module_v1.16.tar.gz注意:修改模块中对应的路径，否则模块不能正确安装加载；
cd fastdfs-nginx-module/srcvi conf #更改如下，去掉local，并指定lib64（64系统）CORE_INCS="$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon/CORE_LIBS="$CORE_LIBS -L/usr/lib64 -lfastcommon -lfdfsclient"(2).nginx 添加fastdfs-nginx-module模块;
ulimit -SHn 102400useradd -s /sbin/nologin nginxtar zxf nginx-1.8.0.tar.gzcd nginx-1.8.0./configure --user=nginx --group=nginx --add-module=../fastdfs-nginx-module/src/ --prefix=/usr/local/nginx --with-http_stub_status_module--with-http_ssl_module --with-http_realip_modulemake && make install(3).nginx 配置fastdfs;
nginx_Fastdfs1 配置;
[root@nginx_Fastdfs1 nginx-1.8.0]# cat /usr/local/nginx/conf/nginx.conf
user nginx;worker_processes 1;events { worker_connections 1024;}http{ server { listen 80;server_name 192.168.1.190;root /webdata/fastdfs.com;index index.html index.htm;location / { if (!-e $request_filename) { rewrite ^(.)$ /index.html last;}location /group1/M00 { root /export/fastdfs/storage/data/;ngx_fastdfs_module;}location ~ ..(gif|jpg|jpeg|png|bmp|swf|js)$ {fastcgi_cache_valid 200 10m; fastcgi_cache_valid 304 3m; fastcgi_cache_valid 301 302 1h; fastcgi_cache_valid any 1m; fastcgi_cache_min_uses 1; fastcgi_cache_use_stale error timeout invalid_header http_500; fastcgi_cache_key $host$request_uri; access_log off; } }access_log off;
}
}nginx_Fastdfs2 配置;
[root@nginx_Fastdfs2]# cat /usr/local/nginx/conf/nginx.conf
events { worker_connections 1024;}http{ server { listen 80;server_name 192.168.1.200;root /webdata/fastdfs.com;index index.html index.htm;location / { if (!-e $request_filename) { rewrite ^(.)$ /index.html last;}location /group1/M00 { root /export/fastdfs/storage/data/;ngx_fastdfs_module;}location ~ ..(gif|jpg|jpeg|png|bmp|swf|js)$ { #fastcgi_cache cache_one; #nginx.conf 开启cache,否则启动报错fastcgi_cache_valid 200 10m;fastcgi_cache_valid 304 3m;fastcgi_cache_valid 301 302 1h;fastcgi_cache_valid any 1m;fastcgi_cache_min_uses 1;fastcgi_cache_use_stale error timeout invalid_header http_500;fastcgi_cache_key $host$request_uri;access_log off;}}access_log off;}}6.fastdfs服务管理;
(1)编写启动脚本;[root@nginx_Fastdfs1]# cat /etc/init.d/fdfs_storaged#!/bin/bash. /etc/init.d/functions
PRG=/usr/local/fastdfs_packages/FastDFS/storage/fdfs_storagedCONF=/etc/fdfs/storage.confif [ ! -f $PRG ]; thenecho "file $PRG does not exist!"exit 2fiif [ ! -f /usr/local/fastdfs_packages/FastDFS/stop.sh ]; thenecho "file /usr/local/fastdfs_packages/FastDFS/stop.sh does not exist!"exit 2fiif [ ! -f /usr/local/fastdfs_packages/FastDFS/restart.sh ]; thenecho "file /usr/local/fastdfs_packages/FastDFS/restart.sh does not exist!"exit 2fiif [ ! -f $CONF ]; thenecho "file $CONF does not exist!"exit 2fiCMD="$PRG $CONF"
RETVAL=0start() {
echo -n $"Starting FastDFS storage server: "$CMD &RETVAL=$?echoreturn $RETVAL}stop() { /usr/local/fastdfs_packages/FastDFS/stop.sh $CMDRETVAL=$?return $RETVAL}rhstatus() { status fdfs_storaged}restart() { /usr/local/fastdfs_packages/FastDFS/restart.sh $CMD &}case "$1" instart)start;;stop)stop;;status)rhstatus;;restart|reload)restart;;condrestart)restart;;*)echo $"Usage: $0 {start|stop|status|restart|condrestart}"exit 1esacexit $?[root@nginx_Fastdfs1]# cat /etc/init.d/fdfs_trackerd #!/bin/bash. /etc/init.d/functions
PRG=/usr/local/fastdfs_packages/FastDFS/tracker/fdfs_trackerdCONF=/etc/fdfs/tracker.confif [ ! -f $PRG ]; then
echo "file $PRG does not exist!"exit 2fiif [ ! -f /usr/local/fastdfs_packages/FastDFS/stop.sh ]; then
echo "file /usr/local/bin/stop.sh does not exist!"exit 2fiif [ ! -f /usr/local/fastdfs_packages/FastDFS/restart.sh ]; then
echo "file /usr/local/bin/restart.sh does not exist!"exit 2fiif [ ! -f $CONF ]; then
echo "file $CONF does not exist!"exit 2fiCMD="$PRG $CONF"
RETVAL=0start() {
echo -n $"Starting FastDFS tracker server: "$CMD &RETVAL=$?echoreturn $RETVAL}stop() { /usr/local/fastdfs_packages/FastDFS/stop.sh $CMDRETVAL=$?return $RETVAL}rhstatus() { status fdfs_trackerd}restart() { /usr/local/fastdfs_packages/FastDFS/restart.sh $CMD &}case "$1" instart)start;;stop)stop;;status)rhstatus;;restart|reload)restart;;condrestart)restart;;*)echo $"Usage: $0 {start|stop|status|restart|condrestart}"exit 1esacexit $?(2).fastdfs相关服务启动;
/etc/init.d/fdfs_trackerd start/etc/init.d/fdfs_storaged start /usr/local/nginx/sbin/nginx（3）查看进程运行情况;
[root@nginx_Fastdfs1 /]# netstat -anplt |grep -E "nginx|fdfs"tcp 0 0 192.168.1.200:23000 0.0.0.0: LISTEN 1945/fdfs_storaged tcp 0 0 192.168.1.200:22122 0.0.0.0: LISTEN 1937/fdfs_trackerd tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1798/nginx tcp 0 0 192.168.1.200:35036 192.168.1.200:22122 ESTABLISHED 1945/fdfs_storaged tcp 0 0 192.168.1.200:22122 192.168.1.200:35036 ESTABLISHED 1937/fdfs_trackerd tcp 0 0 192.168.1.200:43282 192.168.1.190:22122 ESTABLISHED 1945/fdfs_storaged tcp 0 0 192.168.1.200:51407 192.168.1.190:23000 ESTABLISHED 1945/fdfs_storaged tcp 0 0 192.168.1.200:22122 192.168.1.190:44927 ESTABLISHED 1937/fdfs_trackerd tcp 0 0 192.168.1.200:22122 192.168.1.190:40724 ESTABLISHED 1937/fdfs_trackerd tcp 0 0 192.168.1.200:23000 192.168.1.190:54332 ESTABLISHED 1945/fdfs_storaged[root@nginx_Fastdfs2 ~]# netstat -anplt |grep -E "nginx|fdfs"
tcp 0 0 192.168.1.190:22122 0.0.0.0: LISTEN 1603/fdfs_trackerd tcp 0 0 0.0.0.0:80 0.0.0.0: LISTEN 1573/nginx tcp 0 0 192.168.1.190:23000 0.0.0.0:* LISTEN 1612/fdfs_storaged tcp 0 0 192.168.1.190:22122 192.168.1.190:54669 ESTABLISHED 1603/fdfs_trackerd tcp 0 0 192.168.1.190:23000 192.168.1.200:51407 ESTABLISHED 1612/fdfs_storaged tcp 0 0 192.168.1.190:22122 192.168.1.200:43282 ESTABLISHED 1603/fdfs_trackerd tcp 0 0 192.168.1.190:40724 192.168.1.200:22122 ESTABLISHED 1603/fdfs_trackerd tcp 0 0 192.168.1.190:44927 192.168.1.200:22122 ESTABLISHED 1612/fdfs_storaged tcp 0 0 192.168.1.190:54332 192.168.1.200:23000 ESTABLISHED 1612/fdfs_storaged tcp 0 0 192.168.1.190:54669 192.168.1.190:22122 ESTABLISHED 1612/fdfs_storaged7.fastdfs 功能测试;
nginx_Fastdfs1 机器测试;[root@nginx_Fastdfs1]# fdfs_test /etc/fdfs/client.conf upload test.jsp This is FastDFS client test program v5.05Copyright (C) 2008, Happy Fish / YuQingFastDFS may be copied only under the terms of the GNU GeneralPublic License V3, which may be found in the FastDFS source kit.Please visit the FastDFS Home Page for more detail.[2016-06-07 03:04:04] DEBUG - base_path=/tmp, connect_timeout=30, network_timeout=30, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0tracker_query_storage_store_list_without_group: server 1. group_name=, ip_addr=192.168.1.190, port=23000server 2. group_name=, ip_addr=192.168.1.200, port=23000group_name=group1, ip_addr=192.168.1.190, port=23000storage_upload_by_filenamegroup_name=group1, remote_filename=M00/00/00/wKgBvldVySWAObaSAAAAAAAAAAA284.jspsource ip address: 192.168.1.190file timestamp=2016-06-07 03:04:05file size=0file crc32=0example file url: storage_upload_slave_by_filenamegroup_name=group1, remote_filename=M00/00/00/wKgBvldVySWAObaSAAAAAAAAAAA284_big.jspsource ip address: 192.168.1.190file timestamp=2016-06-07 03:04:05file size=0file crc32=0example file url:nginx_Fastdfs2 机器测试;
[root@nginx_Fastdfs2 ~]# fdfs_test /etc/fdfs/client.conf upload test.jspThis is FastDFS client test program v5.05Copyright (C) 2008, Happy Fish / YuQingFastDFS may be copied only under the terms of the GNU GeneralPublic License V3, which may be found in the FastDFS source kit.Please visit the FastDFS Home Page for more detail.[2016-06-07 03:06:33] DEBUG - base_path=/tmp, connect_timeout=30, network_timeout=30, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0tracker_query_storage_store_list_without_group: server 1. group_name=, ip_addr=192.168.1.190, port=23000server 2. group_name=, ip_addr=192.168.1.200, port=23000group_name=group1, ip_addr=192.168.1.200, port=23000storage_upload_by_filenamegroup_name=group1, remote_filename=M00/00/00/wKgByFdVybiAcFDfAAAAAAAAAAA902.jspsource ip address: 192.168.1.200file timestamp=2016-06-07 03:06:32file size=0file crc32=0example file url: storage_upload_slave_by_filenamegroup_name=group1, remote_filename=M00/00/00/wKgByFdVybiAcFDfAAAAAAAAAAA902_big.jspsource ip address: 192.168.1.200file timestamp=2016-06-07 03:06:32file size=0file crc32=0example file url:转载于:https://blog.51cto.com/breaklinux/2150199