Удаление дубликатов предложений

Задача!
Нужно в текстовом файле удалить дубликаты предложений. Фильтр не должен быть чувствительным к регистру. Как фильтр должен определить предложение - начало с большой буквы в конце точка.
Понимаю что, скорее всего придётся заказывать простенький скриптик под эту задачу, но может готовые решения имеются.

примерно так, если брать за внимание что дубликаты полностью идентичны

Код:

<?php $z=fopen("text.txt","w"); fputs($z,join("",array_unique(explode(".",file_get_contents("text.txt")))); fclose($z); ?>

чуть сложнее, если нужно учитывать регистронезависимость:

Код:

<?php function inarray(){   list($str,$a)=func_get_args();   foreach($a as $v){      if(strcasecmp($str, $v)==0){return true;}    }   return false; } $result=array(); foreach (explode(".",file_get_contents("text.txt")) as $k=>$v) {   $v=chop($v);   if(!inarray($v, $result)){$result[$k]=$v;} } $z=fopen("text.txt","w"); fputs($z,join("\n",$result); fclose($z); ?>

само собой на больших текстах это все будет лагать

Код:

<? $input="input.txt"; $output="output.txt"; function del($k,$v){ global $mas; for($i=0;$i<count($mas);$i++){$pat="/".$v."/i"; if (preg_match($pat,$mas[$i]) && $k!=$i){$mas[$i]="";} } } $file=file_get_contents($input); preg_match_all("/([A-Z].+\.)/U",$file,$o); $mas=$o[0]; $rp=preg_replace("/([A-Z].+\.)/U","{#PREDLOZ#}",$file); foreach($mas as $k=>&$v){ if (eregi("^[A-Z]",$v)){del($k,$v);} } foreach ($mas as $v){ $rp=preg_replace("/{#PREDLOZ#}/",$v,$rp,1);} $f=fopen($output,"w");fwrite($f,$rp);fclose($f); ?>

Спасибо! Но оба скрипта выдают ошибку
localhost: parse error
-=Faraon=-: Warning: file_get_contents(input.txt): failed to open stream: No such file or directory in papka/domen/papka/papka/text.php on line 12

Хотя input.txt есть и права на него 777 и output.txt есть с такими же правами.

поправка:
нужна еще одна скобка в строчке:

Код:

fputs($z,join("\n",$result);

то есть должно быть:
fputs($z,join("\n",$result));

COOLJACK писал:

Спасибо! Но оба скрипта выдают ошибку localhost: parse error -=Faraon=-: Warning: file_get_contents(input.txt): failed to open stream: No such file or directory in papka/domen/papka/papka/text.php on line 12 Хотя input.txt есть и права на него 777 и output.txt есть с такими же правами.

права поставь на php файл.

localhost: Как-то стрёмно второй скрипт дубли удалил переколбасило весь текст.
-=Faraon=-: запустил скрипт вначале всё было ок спустя время появилось это
Warning: preg_match(): Unknown modifier '/' in /var/www/1/text2.php on line 8
Warning: preg_match(): Compilation failed: missing ) at offset 56 in /var/www/1/text2.php on line 8
Warning: preg_match(): Unknown modifier 'r' in /var/www/1/text2.php on line 8
Warning: preg_match(): Unknown modifier 'b' in /var/www/1/text2.php on line 8

Ждал где-то час так и не дождался результата всё отрубил. Текс 650 КБ в юникс формате, если это поможет.

приведи пример текста до и после

7 строк после удаления дублей

Код:

10 Things Sex Over 60 Can Teach You About Sex Under 40 Got your attention? Everyone's favorite subject I'll begin by stating again a myth that dies hard: That people over 60 [substitute your own age] aren't having sex Part of this is that we just don't want to think of our parents as actually having sex, no matter how old they are, no matter how old we are Guess what? Your parents are doing it So are your grandparents If they're a healthy couple, that is, who enjoy each other and are still enjoying life

7 строк до удаления дублей

Код:

10 Things Sex Over 60 Can Teach You About Sex Under 40 Got your attention? Everyone's favorite subject. I'll begin by stating again a myth that dies hard: That people over 60 [substitute your own age] aren't having sex. Part of this is that we just don't want to think of our parents as actually having sex, no matter how old they are, no matter how old we are. Guess what? Your parents are doing it. So are your grandparents. If they're a healthy couple, that is, who enjoy each other and are still enjoying life. Sex over 60 is sublime. Sex under 60 is also sublime. In this article let's look at the things that make sex over 60 sublime, in a humorous way, because some of the things that startle, amuse, and slightly embarrass you about your Granny are some of the things that can make sex under 40 sublime. 1. Your Mom arrives at your house and you're having a discussion over breakfast cereal, and she says, ?Monica, I hope you're letting Harry make love to you with the lights on. Men like that, you know.? Put the blush away. She's talking, and that's what makes for good sex. Older folks don't mince words. Get comfortable with anatomy, natural processes, and needs, and talk about them openly. Talking about what ?embarrasses? you could be exactly what could save or enrich your marriage. And be open-minded. Some men like to make love with the lights on, some don't. Some women like to make love with the lights on, some don't. How will you know if you don't ask? How will you know if you don't try it? 2. Granny comes to see your new baby boy and says, ?Looks like his penis is healing just fine, Elberta." Those cute little phrases I won't go into are fine for bonding, but they take something away in the long run. Call your body parts by their real names. This will both empower you, and demystify sex, and that's a great combination. 3. You're appalled at your mother-in-law's lifestyle, and also a bit jealous. Since her husband died 5 years ago, she's been living in a resort retirement community. She plays tennis and golf with her boyfriend, spends lazy afternoons by the pool, and they take long cruises. She looks 10 years younger, has a glow, and your husband says he's never seen her happier. When they come to visit you, they're always hugging and holding hands. You walk around a corner and catch them kissing! You MUST make time to approach this leisurely lifestyle with your partner. You must create a space for what comes naturally to occur, and in the midst of crying babies, promotions and downsizings, leaky faucets, dirty dishes, creeping kudzu, and dog-poop scooping, remember your dating days when you spent hours just gazing into each other's eyes and touching. 4. Your Granpa comes over and says to you, ?Son, you leave that pretty little filly alone like this and she's gonna jump the fence and go lookin'. When's the last time you [wink wink nudge nudge].? The words may make you squirm. The message may too, because it's not about denial. What's going on has been labeled, the consequences outlined, and the solution presented. It works. Try it. Your own style, words, and timing, of course, with knowledge of your wife, not his or anyone else's. 5. Your Mom says, ?Jen, when I walk in here I can cut the tension in the air. There's something you and Nick aren't dealing with.? If you deal with what comes up when it comes up, there won't be tension in the air. Time and experience teach us to read other people's emotions, take the temperature in the air, and become more empathic, which makes for great relationships. You can jumpstart this process by studying Emotional Intelligence with a certified EQ coach. 6. Dinner's over and your parents are leaving. Your father, a man of few words says, ?Anne, your mom and I are coming over next weekend to take care of kids while you and Anthony get away. No arguments.? Make this time. Money is not the crucial component. Put the kids down for Saturday naps and have your own getaway. If there is money, hire a sitter and go to a hotel for a weekend ? somewhere with that resort community atmosphere that's letting your mother-in-law have so much fun. 7. Aunt Grace is over and she's talking about her health again. Staying on top of your physical health is important. Dysfunction and displeasure can have a physical component. Start with a physical checkup if your desire or capacity has faltered. 8. Mimi's sitting on the patio with you. ?I just read the other day,? she says ? Read, study, stay abreast of what's new. There's so much on the Internet that's easy access. 9. ?The secret to a good marriage?? your Uncle Ben is saying, "It's ...." Unless you want a room-mate who does your laundry [mows the lawn], get your marriage and sexual relationship out of the realm of ?mysterious and mystical.? Try telling someone you can't stand garlic in your mashed potatoes without using the words ?mashed potatoes,? ?garlic,? and ?can't stand.? If you have trouble getting started on communicating there are wonderful relationship, marriage and sex coaches, counselors, and therapists ready to help! It's worth it. 10. Take it up a notch. When your six-year-old daughter doesn't like the way you comb her hair, she screams, fidgets, pulls away, cries, pushes you away, runs to her room and slams the door shut, says it makes her head hurt, and learns to tell when it's coming and disappears at these times. You connect the dots. Sex. It does a body good. Deal with it like grown ups do!

как вариант можно сделать проще

сделать по предложению на строку (точка+пробел заменить на точка+перенос строки)
и потом в шеле cat input|sort|uniq > ouput

это если не использовать пхп и прочее

Думаю многим будет полезна данная бесплатная утилита:
Секатор v1.03 - программа для обработки текстовых баз ключевиков.
Функционал программы:
+ Обработка баз (баз ключевиков, ссылок и т.п.) - удаление дублей, рандомизация.
+ Чистка базы - замена и удаление символов.
+ Разбивка базы - разбивка на N частей или на N строк.

http://www.vipraskrutka.net/topic.php?forum=8&topic=6

w84me писал:

как вариант можно сделать проще сделать по предложению на строку (точка+пробел заменить на точка+перенос строки) и потом в шеле cat input|sort|uniq > ouput это если не использовать пхп и прочее

Ухты! А это решение. Даже в винде это можно организовать.
Пасиб!