posted January 26, 2004 11:23 AM            

I've just done 2 things and have a new bug...

a) switched web hosts (hostforweb, running Linux) (last week)
b) enabled HTML-formatting (3 weeks ago)

and I've found several events that went "missing'.

I 'found' them in the events.txt file. The issues seems to be line-feeds.

eg. the file showed up as

652 EventXYZ we are going
\nto the cirus 1600 1800 ..... (edit: realized all of the offending lines start with \n)

If I remove the line-feed (so it looks like)

652 EventXYZ we are going\nto the circus 1600 1800 .....

then everything is ok.

What I am not sure about, is how the line-feed got there in the first place... possibilities that come to mind are

- something in the HTML-formatting code (that was previously disabled)
- something on our web host
- something in the OS/browsers being used by the new officers in our club that are scheduling events for 2004.

Anyone seen this, know more?

Thanks,
Mike

posted January 26, 2004 01:30 PM            

I had to manually rebuild the events.txt data file for one for my clients for pretty much the same reason, lines broken with added line feeds. They haven't moved servers.

The funny thing is that there were also several (numerous!) ^M characters in the file which is often used by Mac computers as line feeds although no one entering data uses a Mac nor is it hosted on a Mac server.

I have not yet found the cause nor has the problem occurred again (yet?). I still have the corrupted data file if Matt can tell anything from it.

Dan O.

posted January 26, 2004 02:14 PM            

The corruption appears (in my case anyway) to be a ^M followed by a return character then on the next line a "\n" text entry. Those 4 characters seem to take the place of where a space should be and all in the 'description' (the only textbox) field in random places (ie. not always at the start/end of the entry).

I'm curious Zim, how many records were in your corrupted events.txt file? This one has 4,380 and am wondering if maybe size has something to do with it (although I believe others with much fewer events has experienced the corruption resulting in no events displaying too).

Dan O.

posted January 26, 2004 03:33 PM            

I had maybe 4 events (out of ~800), where long event descriptions were nicely "formatted", resulting in 8-10 line events (with the line-feeds).

I tried to duplicate the effect, with a test event, entering combinations of HTML BR's, \n, \r, etc into the admin interface and could not reproduce the effect.

I've asked my user if he knows how me might have pulled it off, but no answer yet. I do know that he is a Windows guy, so I think this is not OS specific.

posted January 28, 2004 12:10 PM            

To me this rules out:
- the new web host inserting these line-feeds
- enabling HTML-formatting (in Dec 2003)

Line-length does not seem to be the cause (unless it is browser dependent during submission) (I use Mozilla)

It does not rule something on the new web host getting "upset" about these line-feeds, where the previous host did not, but I assume (need to search the code) that each event is assumed to be one line.

So now I am stumped, but more convinced of this as a general bug.

My 3.2 install is modified only in the template, and an added 90-day "view". Nothing that should touch events.txt.

Mike

------------------

posted January 28, 2004 12:27 PM            

Dan O.

posted January 28, 2004 01:00 PM            

In lib/Event.inc I did note that the search routine seems to expect events to be one single line, and that lines are skipped if they start with a # (comment) or any kind of white-space.

All of my "broken" events have their "extra lines" starting with

\nblah blah blah

but the \n is 2 characters (not a displayed version of a line-feed), so that should be non-white-space. My guess is that the line does not correctly "split" into valid fields, and is then tossed?

The question (in my mind) now is how are these lines getting broken up in events.txt?

Cheers,
Mike

[This message has been edited by Zim (edited January 28, 2004).]

posted January 29, 2004 08:07 AM            

are you able to produce a NEW bad event? I cannot.

It occurs to me that a "patch" should be simple, doing something like

$event_line =~ s/\n//g;
$event_line .= "\n";

remove any linefeeds in the line then tack one on the end (lazy way, I know it could be one statement).

Guess this would be a least 2 places... event create, and event edit.

Seems like it should patch the problem, tho it would not solve why it was happening in the first place. But since I have not found a way to myself create one of these events, I can't for sure say this would fix it.

Cheers,
Mike

posted January 29, 2004 09:14 AM            

the command "add_event" calls addRecord, which calls DBFormat which does

$text =~ s|\n|\\n|sg;
$text =~ s|\t|\\t|sg;
return $text;

edit_event calls updateRecords, which finds a match, inserts the new values, calls flattenRecord, which calls DBFormat, so again any linefeeds should be removed...

Still baffled. Ok, where does a \r linefeed come from? Could those cause a problem?

Mike

------------------

posted January 29, 2004 02:19 PM            

I believe that is what Windows systems add to force a new line, actually I think it adds a combination \r\n instead of just \n like on a Unix system.

** are you able to produce a NEW bad event? **

I haven't tried as it is not my calendar installation and it is in use at the moment.

Dan O.

posted January 29, 2004 03:18 PM            

quote:

---

Processing inputs from TEXTAREA fields - dealing with linefeeds and control-M characters

When processing TEXTAREA form inputs, you'll need to do a search and replace on "control-M" characters (if you're using CGI.pm, it will not take care of this for you). This is most relevant to the Calendar Project, since the "description" field for an event should be a TEXTAREA field. Control-M's are included along with the "\n" linefeed character by some browsers when a form is submitted. They can ruin the format of your text data file if you don't remove them, as they may be interpreted as linefeeds (your results will vary on different operating systems). To avoid any problems, do a search and replace as follows:

$input =~ s/\cM//g;

From there, you'll want to remove all carriage return and newline characters too, as they also can ruin the format of your data file. You'll probably want to replace them with a space, like this:

$input =~ s/\n/ /g;
$input =~ s/\r/ /g;

---

PS. I have a request in to my client to try to find out what browsers are being used which end up with corrupted event data to try to see if there's any correlation.

Dan O.

posted January 30, 2004 02:04 PM            

BTW, write this curious piece of code playing around...

#!/usr/bin/perl

my $string = "A\n B\r +\n C\cM D\n";
print $string;
$string =~ s/\n//g;

print "\n";
print $string;
print "\n";

wacky results

Mike

posted February 06, 2004 09:25 AM            

I just checked and found no new failures (over the past week) in the events.txt that was having the trouble. Best patch-thought so far is to write a script to read in the file and search for ^\n lines, removing them. Then either put that into the read-in code of the calendar script itself, or just put it on a cron job and run it once/twice per day.

Mike

posted February 06, 2004 06:05 PM            

The client reports that the users entering data which ended up creating the problems were using IE version # 6.0.2800.1106

Since I installed the trap for ^M there hasn't been any more corruption of the events.txt file. I'll keep checking for the next weeek or so.

Dan O.

posted February 07, 2004 08:43 AM           

Reading this thread, i went into events.txt and deleted all of the /n characters.

posted February 07, 2004 03:06 PM           

I do notice that those events which are copied directly from another source and pasted in.. seem to have other codings..such as /t inserted..

Events that are directly hand typed, are not experiencing the additional codes.

This problem of disappearing events occurred early in February on one site; and the strange thing is.. the owner is using 3 calendars. All three appear to have every file modified at the same time, same date.. 2/2/2004 3:444pm. That seems as if it might be something from the server.

posted February 07, 2004 06:08 PM            

Trisha, the copy/paste "wrinkle" does strike a cord for me, I need to try that in my attempts to cause a failure. All of my attempts were fresh typing.

Thanks!
Mike

posted February 08, 2004 01:37 PM            

In the sub DBFormat in the DBFile.pm file located in the calendarscript/lib directory.

$text =~ s|\cM| |sg;         # ADDED Line
$text =~ s|\n|\\n|sg;
$text =~ s|\t|\\t|sg;
return $text;
}

There should be NO space between the | | in the added line. This forum software adds it.

Dan O.

------------------

posted February 08, 2004 01:44 PM            

You mean into the regular 'Add Event' form fileds or directly into the database file(s)?

** seem to have other codings.. such as /t inserted.. **

A \t represents a [Tab] keyboard key.

Dan O.

posted February 17, 2004 06:54 PM            

I just checked today, no new corruption. They were experiencing this problem at least every couple of days but there hasn't been any in 2+ weeks (so far).

Dan O.

posted February 17, 2004 07:37 PM            

vim search/replace :%s/\n^\\n/\\n/

I re-saved the file (the global search/replace should have gotten them all this time for sure), and I'll check it when I get back in 2 weeks.

Cheers,
Mike

posted February 21, 2004 01:35 PM            

Try copying first to a text editor (like Windows Notepad, etc.) than copying that data into the CalendarScript form fields. Windows (if you're using it) will sometimes copy hidden code which other applications may use to format text. Putting it into a text editor first should strip the extraneous formatting code.

If they copy Ok after that, you can be sure it isn't a problem with CalendarScript but the way the other applications or operating system is handing the copied data.

** lots of /t's and /n's **

Are you sure they're not \n and \t?

Dan O.

posted March 05, 2004 08:14 AM            

#!/usr/local/bin/perl -w

use MIME::Lite;

my $file = "/xyz/www/cgi-bin/calendarscript/calendars/default/events.txt";

open (EVENTS, "<$file");
$bad = 0;
while ($line = <EVENTS> ){
if (($line !~ /^\d+/) && ($line !~ /^#format/)) {
print "$line\n";
$bad = $bad + 1;
}
}
close (EVENTS);
if ($bad != 0) {
my $msg = new MIME::Lite
From =>'webmaster@xyz.org',
To =>'webmaster@xyz.org',
Subject =>"Calendar events.txt report",
Type =>'TEXT',
Data =>"There are $bad bad lines in the events.txt file that do not start with an event number.\n";
$msg->send;
}

Fingers crossed that we have this solved.

Mike

------------------

posted March 25, 2004 03:12 PM            

Mike

posted March 25, 2004 05:16 PM            

I just checked my client's too and still no recurrence of the problem.

Matt did upgrade that function in the 3.21 version but it doesn't look like he did anything about the ^M characters which I thought was the cause.

JFYI

Dan O.

posted April 13, 2004 04:04 PM         

I mostly us mozilla firefox/bird for entries, but occassionally i have to use a Explorer. so that goes with some of what you say, though for the life of me, i don't know which entries I did in IE.
I have only 340 entires so size doesn't matter.
When i open the tar-ed event.txt in text editor (editplus2) i do see the breaks /n.

I open the original event.txt from the old server and I also see /n but I see the little boxes at the beginning of each entry ( i don't know what they are called but I know it has something to do with line returns)

Anyway.. I fixed my new server event.txt by downloading the old off the server and uploading it to the new one.. so hopefully it will continue to work.

Hope the may help anyone.

------------------