现在有一堆的yum安装包需要安装，于是就想使用批量安装，合并后的安装包有重复的，于是决定先去重
源安装包如下：

python-devel python-imaging zip unzip openssl openssl-devel gcc libxml2 libxml2-dev libxslt* zlib zlib-devel libjpeg-devel libpng-devel libwebp libwebp-devel freetype freetype-devel lsof pcre pcre-devel vixie-cron crontabs icu libicu-devel c-ares make cmake gcc gcc-c++ gcc-g77 flex bison file libtool libtool-libs autoconf kernel-devel patch wget libjpeg libjpeg-devel libpng libpng-devel libpng10 libpng10-devel gd gd-devel libxml2 libxml2-devel zlib zlib-devel glib2 glib2-devel tar bzip2 bzip2-devel libevent libevent-devel ncurses ncurses-devel curl curl-devel libcurl libcurl-devel e2fsprogs e2fsprogs-devel krb5 krb5-devel libidn libidn-devel openssl openssl-devel vim-minimal gettext gettext-devel ncurses-devel gmp-devel pspell-devel libcap diffutils ca-certificates net-tools libc-client-devel psmisc libXpm-devel git-core c-ares-devel libicu-devel libxslt libxslt-devel zip unzip glibc.i686 libstdc++.so.6 cairo-devel bison-devel ncurses-devel libaio-devel perl perl-devel perl-Data-Dumper lsof pcre pcre-devel vixie-cron crontabs expat-devel readline-devel wget python-devel python-imaging zip unzip openssl openssl-devel gcc libxml2 libxml2-dev libxslt* zlib zlib-devel libjpeg-devel libpng-devel libwebp libwebp-devel freetype freetype-devel lsof pcre pcre-devel vixie-cron crontabs crontabs icu libicu-devel c-ares

思路：使用awk更换分隔符，空格转换为换行，排序，去重，转换换行为空格，合并到一行，去除行首的空格

cat yumqucong.sh

#!/bin/bash
#writen by Aiker

echo "python-devel python-imaging zip unzip openssl openssl-devel gcc libxml2 libxml2-dev libxslt* zlib zlib-devel libjpeg-devel libpng-devel libwebp libwebp-devel freetype freetype-devel lsof pcre pcre-devel vixie-cron crontabs icu libicu-devel c-ares make cmake gcc gcc-c++ gcc-g77 flex bison file libtool libtool-libs autoconf kernel-devel patch wget libjpeg libjpeg-devel libpng libpng-devel libpng10 libpng10-devel gd gd-devel libxml2 libxml2-devel zlib zlib-devel glib2 glib2-devel tar bzip2 bzip2-devel libevent libevent-devel ncurses ncurses-devel curl curl-devel libcurl libcurl-devel e2fsprogs e2fsprogs-devel krb5 krb5-devel libidn libidn-devel openssl openssl-devel vim-minimal gettext gettext-devel ncurses-devel gmp-devel pspell-devel libcap diffutils ca-certificates net-tools libc-client-devel psmisc libXpm-devel git-core c-ares-devel libicu-devel libxslt libxslt-devel zip unzip glibc.i686 libstdc++.so.6 cairo-devel bison-devel ncurses-devel libaio-devel perl perl-devel perl-Data-Dumper lsof pcre pcre-devel vixie-cron crontabs expat-devel readline-devel wget python-devel python-imaging zip unzip openssl openssl-devel gcc libxml2 libxml2-dev libxslt* zlib zlib-devel libjpeg-devel libpng-devel libwebp libwebp-devel freetype freetype-devel lsof pcre pcre-devel vixie-cron crontabs crontabs icu libicu-devel c-ares" | awk '{for(x=1;x<=NF;x++) {print $x}} {print "\n"}' | sort | uniq | tr "\r\n" " " | sed -e 's/\r\n/ /g' -e 's/^ //g'

输出：

autoconf bison bison-devel bzip2 bzip2-devel ca-certificates cairo-devel c-ares c-ares-devel cmake crontabs curl curl-devel diffutils e2fsprogs e2fsprogs-devel expat-devel file flex freetype freetype-devel gcc gcc-c++ gcc-g77 gd gd-devel gettext gettext-devel git-core glib2 glib2-devel glibc.i686 gmp-devel icu kernel-devel krb5 krb5-devel libaio-devel libcap libc-client-devel libcurl libcurl-devel libevent libevent-devel libicu-devel libidn libidn-devel libjpeg libjpeg-devel libpng libpng10 libpng10-devel libpng-devel libstdc++.so.6 libtool libtool-libs libwebp libwebp-devel libxml2 libxml2-dev libxml2-devel libXpm-devel libxslt libxslt* libxslt-devel lsof make ncurses ncurses-devel net-tools openssl openssl-devel patch pcre pcre-devel perl perl-Data-Dumper perl-devel psmisc pspell-devel python-devel python-imaging readline-devel tar unzip vim-minimal vixie-cron wget zip zlib zlib-devel

也可以将最初的yum包保存为文件，然后对文件进行处理：

cat yumlist.txt

python-devel python-imaging zip unzip openssl openssl-devel gcc libxml2 libxml2-dev libxslt* zlib zlib-devel libjpeg-devel libpng-devel libwebp libwebp-devel freetype freetype-devel lsof pcre pcre-devel vixie-cron crontabs icu libicu-devel c-ares make cmake gcc gcc-c++ gcc-g77 flex bison file libtool libtool-libs autoconf kernel-devel patch wget libjpeg libjpeg-devel libpng libpng-devel libpng10 libpng10-devel gd gd-devel libxml2 libxml2-devel zlib zlib-devel glib2 glib2-devel tar bzip2 bzip2-devel libevent libevent-devel ncurses ncurses-devel curl curl-devel libcurl libcurl-devel e2fsprogs e2fsprogs-devel krb5 krb5-devel libidn libidn-devel openssl openssl-devel vim-minimal gettext gettext-devel ncurses-devel gmp-devel pspell-devel libcap diffutils ca-certificates net-tools libc-client-devel psmisc libXpm-devel git-core c-ares-devel libicu-devel libxslt libxslt-devel zip unzip glibc.i686 libstdc++.so.6 cairo-devel bison-devel ncurses-devel libaio-devel perl perl-devel perl-Data-Dumper lsof pcre pcre-devel vixie-cron crontabs expat-devel readline-devel wget python-devel python-imaging zip unzip openssl openssl-devel gcc libxml2 libxml2-dev libxslt* zlib zlib-devel libjpeg-devel libpng-devel libwebp libwebp-devel freetype freetype-devel lsof pcre pcre-devel vixie-cron crontabs crontabs icu libicu-devel c-ares

cat yumqucong.sh

#!/bin/bash
#writen by Aiker

awk '{for(x=1;x<=NF;x++) {print $x}} {print "\n"}' yumlist.txt | sort | uniq | tr "\r\n" " " | sed -e 's/\r\n/ /g' -e 's/^ //g'