Check-in [1de0cb1b8f]

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Merge branch 'master' of etoilebsd.net:cplanet

Conflicts: Makefile cplanet.c

Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 1de0cb1b8f6c192fabe22040cd2f868fd87e152e
User & Date: bapt 2010-12-02 20:43:30
Context
2010-12-03
15:19
Finish killing libnxml/libmrss + fix atom, rss and xhtml1.1 template check-in: 8b6550d288 user: bapt tags: trunk
2010-12-02
20:43
Merge branch 'master' of etoilebsd.net:cplanet

Conflicts: Makefile cplanet.c check-in: 1de0cb1b8f user: bapt tags: trunk

20:28
First attempt at removing libnxml libmrss replaced by curl/expat check-in: a7def9edf7 user: bapt tags: trunk
2010-03-01
12:28
Fix compilation with Werror and libiconv vs POSIX iconv(3) implementation check-in: 5d98ab1f44 user: bapt tags: trunk
Changes

Changes to COPYING.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
            DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
                    Version 2, December 2004

 Copyright (C) 2004 Sam Hocevar
  14 rue de Plaisance, 75014 Paris, France
 Everyone is permitted to copy and distribute verbatim or modified
 copies of this license document, and changing it is allowed as long
 as the name is changed.

            DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

  0. You just DO WHAT THE FUCK YOU WANT TO.

|
|
|
|
|
|
|
|
|
|
|
|
|
<
1
2
3
4
5
6
7
8
9
10
11
12
13

Copyright (c) 2010, Baptiste Daroussin

Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

Changes to Makefile.

1
2
3

4


5



6
7






















PROG=	cplanet
SRCS=	cplanet.c cplanet.h
CFLAGS+=	-Wall -g -I/usr/local/include -I/usr/local/include/ClearSilver

LDADD+=	-L/usr/local/lib -lmrss -lnxml -lz -lneo_cs -lneo_utl -lneo_cgi -liconv -lexpat -lcurl







.include <bsd.prog.mk>






















|
<
|
>
|
>
>

>
>
>

<
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1

2
3
4
5
6
7
8
9
10
11

12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
CC=		gcc

INCLUDES=	-I/usr/local/include -I/usr/local/include/ClearSilver
LIBDIR=		-L/usr/local/lib
LIBS=		-lmrss -lnxml -lz -lneo_cs -lneo_utl -lneo_cgi -liconv -lexpat -lcurl
LDFLAGS+=	${LIBDIR}
CFLAGS+=	-Wall -Werror -pipe -O2

PROG=		cplanet
SRCS=		cplanet.c
OBJS=		${SRCS:.c=.o}


PREFIX?=	/usr/local
BINDIR?=	${PREFIX}/bin
MANDIR?=	${PREFIX}/man/man1

all: ${PROG}

${PROG}: ${OBJS}
	${CC} ${LDFLAGS} ${CFLAGS} ${INCLUDES} ${OBJS} -o $@ ${LIBS}

.c.o:
	${CC} -o $@ -c $< ${CFLAGS} ${INCLUDES}

install:
	install -o root -g wheel -m 755 ${PROG} ${BINDIR}
	install -o root -g wheel -m 755 ${PROG}.1 ${MANDIR}

deinstall:
	rm -f ${BINDIR}/${PROG}
	rm -f ${MANDIR}/${PROG}.1

clean:
	rm -f ${PROG} *.o *.core

Changes to TODO.

1
2
3
4
5
6
7
- generate RSS using clearsilver template
- generate OPML using clearsilver template
- generate ATOM using cleansilver
- add an option to log in syslog (for crontab users)
- check every kind of errors : file not found, etc.
- recheck all the free() and *_free() to avoid memory leaks
- add info, verbose and debug flags
<
<
<
<
<
<
<














Changes to cplanet.1.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46



.TH CPlanet 1 "Jul 17, 2009"
.SH NAME
\fBCPlanet\fP \- a RSS Feed Aggregator to generate a planet
.SH SYNOPSIS
\fBcplanet\fP -c config.hdf [-l]
.SH DESCRIPTION
\fBCPlanet\fP is a RSS/ATOM feed aggregator written in C that generate static html
files.
.SS OPTIONS
.TP
\fB-c config.hdf\fP
specify the path of th hdf configuration file
.TP
\fB-l\fP
set the log to be written in syslog
.SS HDF format
cplanet uses the clearsilver's hdf format as a configuration file, some keys are 
mandatory. Users can add as much keys as they want as they can be used in the
template file.
.TP
\fBCPlanet.Output.N.Path\fP 
N represents a number starting from 0, this is the patch for the file N to
generate
.TP
\fBCPlanet.Output.N.TemplatePath\fP
Path of the template to use to generate the file number N
.TP
\fBCPlanet.Days\fP
oldest post from the feeds to keep in days.
.TP
\fBCPlanet.Feed.N.Name\fP
name of the N feed to fetch
.TP
\fBCPlanet.Feed.N.URL\fP
URL of the N Feed (ATOM or RSS) to fetch.
.br
.br
for more informations concerning the HDF file format please refer to the
clearsilver documentation: http://www.clearsilver.net/docs/man_hdf.hdf
.SS Template File
\fBCPlanet\fP uses the clearsilver template system, please refer to the
clearsilver documentation: http://www.clearsilver.net/docs/man_templates.hdf
.SH AUTHORS
Baptiste Daroussin
.SH CONTRIBUTORS
Freddy DISSAUX



|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
.Dd Feb 22, 2010
.Dt cplanet 1
.Os
.Sh NAME
.Nm cplanet
.Nd RSS Feed Aggregator that generate static html
.Sh SYNOPSIS
.Nm
.Op Fl l
.Fl c Ar config.hdf
.Sh DESCRIPTION
.Nm 
is a RSS/ATOM feed aggregator written in C that generate static html files.
.Sh OPTIONS
.Bl -tag -width indent
.It Fl c Ar config.hdf
specify the path to the hdf configuration file
.It Fl l
set the log to be written in syslog, otherwise log are written to stderr
.El
.Sh CONFIGURATION FILE
.Pp
cplanet uses the clearsilver's hdf format as a configuration file, some keys are 
mandatory. Users can add as much keys as they want as they can be used in the
template file.
.Bl -tag -width indent
.It Ar CPlanet.Output.N.Path
N represents a number starting from 0, this is the patch for the file N to
generate
.It Ar CPlanet.Output.N.TemplatePath
Path of the template to use to generate the file number N
.It Ar CPlanet.Days
oldest post from the feeds to keep in days.
.It Ar CPlanet.Feed.N.Name
name of the N feed to fetch
.It Ar CPlanet.Feed.N.URL
URL of the N Feed (ATOM or RSS) to fetch.
.El
.Pp
for more informations concerning the HDF file format please refer to the
clearsilver documentation: http://www.clearsilver.net/docs/man_hdf.hdf
.Sh TEMPLATE FILE
.Nm
uses the clearsilver template system, please refer to the
clearsilver documentation: http://www.clearsilver.net/docs/man_templates.hdf
.Sh AUTHORS
Baptiste Daroussin
.Sh CONTRIBUTORS
Freddy DISSAUX

Changes to cplanet.c.

1
2
3

4
5










6
7
8
9
10
11
12
...
369
370
371
372
373
374
375
376































377
378
379
380
381
382
383






384
385
386
387

388
389
390
391
392

393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
...
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
...
441
442
443
444
445
446
447



448

449
450
451
452
453
454
455
...
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
...
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586




587




588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
...
640
641
642
643
644
645
646
647
648

649
650





651

652
653

654

655
656

657
658
659
660
661
662
663
664

665
666
667
668
669
670
671
...
682
683
684
685
686
687
688

689
690
691
692
693
694
695
...
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
















































754
755
756
757
758
759
760
761
762
763
764
765
/* This program is free software. It comes without any warranty, to
 * the extent permitted by applicable law. You can redistribute it
 * and/or modify it under the terms of the Do What The Fuck You Want

 * To Public License, Version 2, as published by Sam Hocevar. See
 * http://sam.zoy.org/wtfpl/COPYING for more details. */ 











/* vim:set ts=4 sw=4 sts=4: */

#include <expat.h>
#include <curl/curl.h>
#include <stdbool.h>
#include <sys/queue.h>
................................................................................
		stm.tm_year -= 1900;
		stm.tm_mon -= 1;
		strftime (datebuf, sizeof (datebuf), "%a, %d %b %Y %H:%M:%S %z", &stm);
		return strdup(datebuf);
	}
	return NULL;
}
































/* convert RFC822 to epoch time */
static time_t
str_to_time_t(char *s)
{
	struct tm date;
	time_t t;
	errno = 0;






	char *pos = strptime(s, "%a, %d %b %Y %T", &date);
	if (pos == NULL) {
		errno = EINVAL;
		cplanet_err(1, "Convert '%s' to struct tm failed", s);

	}
	t = mktime(&date);
	if (t == (time_t)-1) {
		errno = EINVAL;
		cplanet_err(1, "Convert struct tm (from '%s') to time_t failed", s);

	}

	return t;
}

/* sort posts by date */

static int
sort_obj_by_date(const void *a, const void *b) {
	HDF **ha = (HDF **)a;
	HDF **hb = (HDF **)b;
	time_t atime = str_to_time_t(hdf_get_valuef(*ha, "Date"));
	time_t btime = str_to_time_t(hdf_get_valuef(*hb, "Date"));

	return (btime - atime);
}

/* prepare the string to be written to the html file */

static NEOERR *
cplanet_output (void *ctx, char *s)
................................................................................
	size_t outsize;
	size_t ret;
	char *output;
	if (!strcasecmp(source_encoding, "UTF-8"))
		return str;

	iconv_t conv;
	const char * inptr = str;
	conv = iconv_open("UTF-8", source_encoding);
	if (conv == (iconv_t)-1) {
		cplanet_warn("Conversion from '%s' to UTF-8 not available", source_encoding);
		return str;
	}
	if (str == NULL)
		return str;
................................................................................
	insize = strlen(str);
	outsize = 4 * insize + 2;
	output = malloc(outsize);
	if ( output == NULL )
		cplanet_err(ENOMEM, "malloc");
	memset(output, 0, outsize);
	char *outputptr = output;



	ret = iconv(conv, (const char**) &inptr, &insize, &outputptr, &outsize);

	if (ret == (size_t)-1) {
		cplanet_warn("Conversion Failed");
		free(output);
		iconv_close(conv);
		return str;
	}
	iconv_close(conv);
................................................................................
	time(&t_now);
	SLIST_FOREACH_SAFE(post, &feed.entries, next, posttemp) {
		if (post->date == NULL) {
			SLIST_REMOVE(&feed.entries, post, post, next);
			continue;
		}

		t_comp = str_to_time_t(post->date);
		if (t_now - t_comp < days) {
			cp_set_name(hdf_dest, pos, hdf_get_valuef(hdf_cfg, "Name"));
			cp_set_feedname(hdf_dest, pos, str_to_UTF8(feed.encoding, feed.blog_title));

			if (post->author != NULL)
				cp_set_author(hdf_dest, pos, str_to_UTF8(feed.encoding, post->author));

................................................................................
{
	mrss_t *feed;
	mrss_error_t ret;
	mrss_item_t *item;
	CURLcode code;
	char *encoding;
	char *name = hdf_get_valuef(hdf_cfg, "Name");
	char *date_format = hdf_get_valuef(hdf_dest, "CPlanet.DateFormat");
	ret = mrss_parse_url_with_options_and_error(hdf_get_valuef(hdf_cfg, "URL"), &feed, NULL, &code);
	if (ret) {
		cplanet_warn("MRSS return error: %s, %s\n",
				ret ==
				MRSS_ERR_DOWNLOAD ? mrss_curl_strerror(code) :
				mrss_strerror(ret), name);
		return pos;
	}
	encoding = feed->encoding;
	for (item = feed->item; item; item = item->next) {
		time_t t_now, t_comp;
		t_comp = str_to_time_t(item->pubDate);




		printf("%s\n", item->pubDate);




		time(&t_now);
		if(t_now - t_comp >= days)
			continue;
		mrss_category_t *tags;
		mrss_tag_t *content; 
		cp_set_name(hdf_dest, pos, name);
		cp_set_feedname(hdf_dest, pos, str_to_UTF8(encoding, feed->title));
		if (item->author != NULL)
			cp_set_author(hdf_dest, pos, str_to_UTF8(encoding, item->author));
		cp_set_title(hdf_dest, pos, str_to_UTF8(encoding, item->title));
		cp_set_link(hdf_dest, pos, item->link);
		cp_set_date(hdf_dest, pos, item->pubDate);
		if (date_format != NULL) {
			char formated_date[256];
			struct tm *ptr;
			time_t time;
			time = str_to_time_t(item->pubDate);
			ptr = localtime(&time);
			strftime(formated_date, 256, date_format, ptr);
			cp_set_formated_date(hdf_dest, pos, formated_date);
		}
		/* Description is only for description tag, we want content if exists */
		if (feed->version == MRSS_VERSION_ATOM_0_3 || feed->version == MRSS_VERSION_ATOM_1_0) {
			if ((mrss_search_tag(item, "content", W3_ATOM, &content) == MRSS_OK && content) || 
					(mrss_search_tag(item, "content", PURL_ATOM, &content) == MRSS_OK && content)) {
				cp_set_description(hdf_dest, pos, str_to_UTF8(encoding, content->value));
			} else {
				cp_set_description(hdf_dest, pos, str_to_UTF8(encoding, item->description));
................................................................................
generate_file(HDF *output_hdf, HDF *hdf)
{
	NEOERR *neoerr;
	CSPARSE *parse;
	STRING cs_output_data;
	string_init(&cs_output_data);
	neoerr = cs_init(&parse, hdf);
	if (neoerr != STATUS_OK) 
		goto warn1;

	char *cs_output = hdf_get_valuef(output_hdf, "Path");
	char *cs_path = hdf_get_valuef(output_hdf, "TemplatePath");





	neoerr = cs_parse_file(parse, cs_path);

	if (neoerr != STATUS_OK) 
		goto warn0;

	neoerr = cs_render(parse, &cs_output_data, cplanet_output);

	if (neoerr != STATUS_OK) 
		goto warn0;

	FILE *output = fopen(cs_output, "w+");
	if (output == NULL)
		cplanet_err(1, "%s", cs_output);
	fprintf(output, "%s", cs_output_data.buf);
	fflush(output);
	fclose(output);
	cs_destroy(&parse);
	string_clear(&cs_output_data);

	return;

warn0:
	cs_destroy(&parse);
warn1:
	string_clear(&cs_output_data);
	nerr_error_string(neoerr,&neoerr_str);
................................................................................
int 
main (int argc, char *argv[])
{
	NEOERR *neoerr;
	HDF *hdf;
	HDF *feed_hdf;
	HDF *output_hdf;

	int pos = 0;
	int days = 0;
	int ch = 0;
	char *hdf_file = NULL;
	if (argc == 1)
		usage();
	while ((ch = getopt(argc, argv, "c:lh")) != -1)
................................................................................

	string_init(&neoerr_str);
	neoerr = hdf_init(&hdf);
	if (neoerr != STATUS_OK) {
		nerr_error_string(neoerr, &neoerr_str);
		cplanet_err(-1, neoerr_str.buf);
	}
	neoerr = hdf_init(&feed_hdf);

	/* Read the hdf file */
	neoerr = hdf_read_file(hdf, hdf_file);
	if (neoerr != STATUS_OK) {
		nerr_error_string(neoerr, &neoerr_str);
		cplanet_err(-1, neoerr_str.buf);
	}
	cp_set_version(hdf);

	char genDate[256];
	struct tm *ptr;
	time_t lt;
	lt = time(NULL);
	ptr = localtime(&lt);
	strftime(genDate, 256, "%a, %d %b %Y %H:%M:%S %z", ptr);
	hdf_set_valuef(hdf, CP_GEN_DATE, genDate);

	days = hdf_get_int_value(hdf,"CPlanet.Days",0);
	days=days *  24 * 60 * 60;

	HDF_FOREACH(feed_hdf,hdf,"CPlanet.Feed.0") {
//		pos = get_posts(feed_hdf, hdf, pos, days);
		pos = fetch_posts(feed_hdf, hdf, pos, days);
	}

	hdf_sort_obj(hdf_get_obj(hdf, "CPlanet.Posts"), sort_obj_by_date);

	/* get every output set in the hdf file and generate them */
	HDF_FOREACH(output_hdf,hdf,"CPlanet.Output.0") {
















































		generate_file(output_hdf, hdf);
	}

	hdf_destroy(&hdf);
	hdf_destroy(&feed_hdf);
	hdf_destroy(&output_hdf);

	if (syslog_flag)
		closelog();

	return EXIT_SUCCESS;
}
|
|
<
>
|
|
>
>
>
>
>
>
>
>
>
>







 








>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

|
|




>
>
>
>
>
>



|
>




|
>







|



|
|
<







 







<







 







>
>
>
|
>







 







|







 







<











|
>
>
>
>
|
>
>
>
>











|
<
<
<
<
<
<
<
<
<







 







|

>


>
>
>
>
>

>
|

>

>
|

>








>







 







>







 







<
<








<
<
<
<
<
<
<
<












>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>




<
<






1
2

3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
...
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454

455
456
457
458
459
460
461
...
474
475
476
477
478
479
480

481
482
483
484
485
486
487
...
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
...
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
...
618
619
620
621
622
623
624

625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657









658
659
660
661
662
663
664
...
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
...
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
...
778
779
780
781
782
783
784


785
786
787
788
789
790
791
792








793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856


857
858
859
860
861
862
/*
 * Copyright (c) 2010, Baptiste Daroussin

 *
 * Permission to use, copy, modify, and distribute this software for any
 * purpose with or without fee is hereby granted, provided that the above
 * copyright notice and this permission notice appear in all copies.
 *
 * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
 * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
 * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
 * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
 * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
 * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
 * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 */

/* vim:set ts=4 sw=4 sts=4: */

#include <expat.h>
#include <curl/curl.h>
#include <stdbool.h>
#include <sys/queue.h>
................................................................................
		stm.tm_year -= 1900;
		stm.tm_mon -= 1;
		strftime (datebuf, sizeof (datebuf), "%a, %d %b %Y %H:%M:%S %z", &stm);
		return strdup(datebuf);
	}
	return NULL;
}

/* convert the iso format as the RFC3339 is a subset of it */
time_t
iso8601_to_time_t(char *s)
{
	struct tm date;
	time_t t;
	errno = 0;
	char *pos = strptime(s, "%Y-%m-%dT%H:%M:%S.%fZ", &date);
	if (pos == NULL) {
		/* Modify the last HH:MM to HHMM if necessary */
		if (s[strlen(s) - 3] == ':' ) {
			s[strlen(s) - 3] = s[strlen(s) - 2];
			s[strlen(s) - 2] = s[strlen(s) - 1];
			s[strlen(s) - 1] = '\0';
		}
		pos =strptime(s, "%Y-%m-%dT%H:%M:%S%z", &date);
	}
	if (pos == NULL) {
		errno = EINVAL;
		cplanet_warn("Convert  ISO8601 '%s' to struct tm failed", s);
		return 0;
	}
	t = mktime(&date);
	if (t == (time_t)-1) {
		errno = EINVAL;
		cplanet_warn("Convert struct tm (from '%s') to time_t failed", s);
		return 0;
	}
	return t;
}

/* convert RFC822 to epoch time */
time_t
rfc822_to_time_t(char *s)
{
	struct tm date;
	time_t t;
	errno = 0;

	if (s == NULL) {
		cplanet_warn("Invalide empty date");
		return 0;
	}

	char *pos = strptime(s, "%a, %d %b %Y %T", &date);
	if (pos == NULL) {
		errno = EINVAL;
		cplanet_warn("Convert RFC822 '%s' to struct tm failed", s);
		return 0;
	}
	t = mktime(&date);
	if (t == (time_t)-1) {
		errno = EINVAL;
		cplanet_warn("Convert struct tm (from '%s') to time_t failed", s);
		return 0;
	}

	return t;
}

/* sort posts by date */

int
sort_obj_by_date(const void *a, const void *b) {
	HDF **ha = (HDF **)a;
	HDF **hb = (HDF **)b;
	time_t atime = hdf_get_int_value(*ha, "Date",0);
	time_t btime = hdf_get_int_value(*hb, "Date",0);

	return (btime - atime);
}

/* prepare the string to be written to the html file */

static NEOERR *
cplanet_output (void *ctx, char *s)
................................................................................
	size_t outsize;
	size_t ret;
	char *output;
	if (!strcasecmp(source_encoding, "UTF-8"))
		return str;

	iconv_t conv;

	conv = iconv_open("UTF-8", source_encoding);
	if (conv == (iconv_t)-1) {
		cplanet_warn("Conversion from '%s' to UTF-8 not available", source_encoding);
		return str;
	}
	if (str == NULL)
		return str;
................................................................................
	insize = strlen(str);
	outsize = 4 * insize + 2;
	output = malloc(outsize);
	if ( output == NULL )
		cplanet_err(ENOMEM, "malloc");
	memset(output, 0, outsize);
	char *outputptr = output;
#ifdef _LIBICONV_H
	ret = iconv(conv, (const char **) &str, &insize, &outputptr, &outsize);
#else
	ret = iconv(conv, &str, &insize, &outputptr, &outsize);
#endif
	if (ret == (size_t)-1) {
		cplanet_warn("Conversion Failed");
		free(output);
		iconv_close(conv);
		return str;
	}
	iconv_close(conv);
................................................................................
	time(&t_now);
	SLIST_FOREACH_SAFE(post, &feed.entries, next, posttemp) {
		if (post->date == NULL) {
			SLIST_REMOVE(&feed.entries, post, post, next);
			continue;
		}

		t_comp = rfc822_to_time_t(post->date);
		if (t_now - t_comp < days) {
			cp_set_name(hdf_dest, pos, hdf_get_valuef(hdf_cfg, "Name"));
			cp_set_feedname(hdf_dest, pos, str_to_UTF8(feed.encoding, feed.blog_title));

			if (post->author != NULL)
				cp_set_author(hdf_dest, pos, str_to_UTF8(feed.encoding, post->author));

................................................................................
{
	mrss_t *feed;
	mrss_error_t ret;
	mrss_item_t *item;
	CURLcode code;
	char *encoding;
	char *name = hdf_get_valuef(hdf_cfg, "Name");

	ret = mrss_parse_url_with_options_and_error(hdf_get_valuef(hdf_cfg, "URL"), &feed, NULL, &code);
	if (ret) {
		cplanet_warn("MRSS return error: %s, %s\n",
				ret ==
				MRSS_ERR_DOWNLOAD ? mrss_curl_strerror(code) :
				mrss_strerror(ret), name);
		return pos;
	}
	encoding = feed->encoding;
	for (item = feed->item; item; item = item->next) {
		time_t t_now, t_comp;
		if (item->pubDate == NULL) {
			cplanet_warn("Invalid date format in the %s feed: skeeping", hdf_get_valuef(hdf_cfg, "Name"));
			continue;
		}
		/* when the feed is not corrupted, mrss stores it in RFC822 format */
		t_comp = rfc822_to_time_t(item->pubDate);
		if (t_comp == 0) {
			cplanet_warn("Invalid date format in the %s feed: skeeping", hdf_get_valuef(hdf_cfg, "Name"));
			continue;
		}
		time(&t_now);
		if(t_now - t_comp >= days)
			continue;
		mrss_category_t *tags;
		mrss_tag_t *content; 
		cp_set_name(hdf_dest, pos, name);
		cp_set_feedname(hdf_dest, pos, str_to_UTF8(encoding, feed->title));
		if (item->author != NULL)
			cp_set_author(hdf_dest, pos, str_to_UTF8(encoding, item->author));
		cp_set_title(hdf_dest, pos, str_to_UTF8(encoding, item->title));
		cp_set_link(hdf_dest, pos, item->link);
		/*cp_set_date(hdf_dest, pos, (long long int)t_comp);*/









		/* Description is only for description tag, we want content if exists */
		if (feed->version == MRSS_VERSION_ATOM_0_3 || feed->version == MRSS_VERSION_ATOM_1_0) {
			if ((mrss_search_tag(item, "content", W3_ATOM, &content) == MRSS_OK && content) || 
					(mrss_search_tag(item, "content", PURL_ATOM, &content) == MRSS_OK && content)) {
				cp_set_description(hdf_dest, pos, str_to_UTF8(encoding, content->value));
			} else {
				cp_set_description(hdf_dest, pos, str_to_UTF8(encoding, item->description));
................................................................................
generate_file(HDF *output_hdf, HDF *hdf)
{
	NEOERR *neoerr;
	CSPARSE *parse;
	STRING cs_output_data;
	string_init(&cs_output_data);
	neoerr = cs_init(&parse, hdf);
	if (neoerr != STATUS_OK)
		goto warn1;

	char *cs_output = hdf_get_valuef(output_hdf, "Path");
	char *cs_path = hdf_get_valuef(output_hdf, "TemplatePath");
	neoerr = cgi_register_strfuncs(parse);

	if (neoerr != STATUS_OK)
		goto warn0;

	neoerr = cs_parse_file(parse, cs_path);

	if (neoerr != STATUS_OK)
		goto warn0;

	neoerr = cs_render(parse, &cs_output_data, cplanet_output);

	if (neoerr != STATUS_OK)
		goto warn0;

	FILE *output = fopen(cs_output, "w+");
	if (output == NULL)
		cplanet_err(1, "%s", cs_output);
	fprintf(output, "%s", cs_output_data.buf);
	fflush(output);
	fclose(output);
	cs_destroy(&parse);
	string_clear(&cs_output_data);

	return;

warn0:
	cs_destroy(&parse);
warn1:
	string_clear(&cs_output_data);
	nerr_error_string(neoerr,&neoerr_str);
................................................................................
int 
main (int argc, char *argv[])
{
	NEOERR *neoerr;
	HDF *hdf;
	HDF *feed_hdf;
	HDF *output_hdf;
	HDF *post_hdf;
	int pos = 0;
	int days = 0;
	int ch = 0;
	char *hdf_file = NULL;
	if (argc == 1)
		usage();
	while ((ch = getopt(argc, argv, "c:lh")) != -1)
................................................................................

	string_init(&neoerr_str);
	neoerr = hdf_init(&hdf);
	if (neoerr != STATUS_OK) {
		nerr_error_string(neoerr, &neoerr_str);
		cplanet_err(-1, neoerr_str.buf);
	}


	/* Read the hdf file */
	neoerr = hdf_read_file(hdf, hdf_file);
	if (neoerr != STATUS_OK) {
		nerr_error_string(neoerr, &neoerr_str);
		cplanet_err(-1, neoerr_str.buf);
	}
	cp_set_version(hdf);









	days = hdf_get_int_value(hdf,"CPlanet.Days",0);
	days=days *  24 * 60 * 60;

	HDF_FOREACH(feed_hdf,hdf,"CPlanet.Feed.0") {
//		pos = get_posts(feed_hdf, hdf, pos, days);
		pos = fetch_posts(feed_hdf, hdf, pos, days);
	}

	hdf_sort_obj(hdf_get_obj(hdf, "CPlanet.Posts"), sort_obj_by_date);

	/* get every output set in the hdf file and generate them */
	HDF_FOREACH(output_hdf,hdf,"CPlanet.Output.0") {
		/* format the date according to the output type */
		char *type = hdf_get_value(output_hdf, "Type", "HTML");
		/* for the posts */
		HDF_FOREACH(post_hdf, hdf, "CPlanet.Posts.0") {
			if(strcasecmp(type, "RSS") == 0) {
				setlocale(LC_ALL, "C");
				time_t date = (time_t)hdf_get_int_value(post_hdf, "Date", time(NULL));
				char formated_date[256];
				/* format to RFC822 */
				struct tm *ptr = gmtime(&date);
				strftime(formated_date, 256, "%a, %d %b %Y %H:%M:%S %z", ptr);
				hdf_set_valuef(post_hdf, "FormatedDate=%s", formated_date);
			} else if (strcasecmp(type, "ATOM") == 0 ){
				time_t posttime = hdf_get_int_value(post_hdf, "Date", time(NULL));
				struct tm *date = gmtime(&posttime);
				hdf_set_valuef(post_hdf, "FormatedDate=%04d-%02d-%02dT%02d:%02d:%02dZ",
						date->tm_year + 1900, date->tm_mon + 1, date->tm_mday,
						date->tm_hour, date->tm_min, date->tm_sec);
			} else {
				/* case html */
				char *date_format = hdf_get_value(hdf, "CPlanet.DateFormat", "%F at %T");
				time_t posttime = hdf_get_int_value(hdf, "Date", time(NULL));
				char formated_date[256];
				struct tm *ptr;
				ptr = localtime(&posttime);
				strftime(formated_date, 256, date_format, ptr);
				hdf_set_valuef(post_hdf, "FormatedDate=%s", formated_date);
			}
		}
		/* the the publication date */
		char genDate[256];
		time_t lt;
		struct tm *ptr;
		lt = time(NULL);
		ptr = gmtime(&lt);
		if (strcasecmp(type, "RSS") == 0) {
			strftime(genDate, 256, "%a, %d %b %Y %H:%M:%S %z", ptr);
			hdf_set_valuef(hdf, CP_GEN_DATE, genDate);
		} else if (strcasecmp(type, "ATOM") == 0) {
			hdf_set_valuef(hdf,"CPlanet.GenerationDate=%04d-%02d-%02dT%02d:%02d:%02dZ",
					ptr->tm_year + 1900, ptr->tm_mon + 1, ptr->tm_mday,
					ptr->tm_hour, ptr->tm_min, ptr->tm_sec);
		} else {
			char *date_format = hdf_get_value(hdf, "CPlanet.DateFormat", "%F at %T");
			ptr = localtime(&lt);
			strftime(genDate, 256, date_format, ptr);
			hdf_set_valuef(hdf, CP_GEN_DATE, genDate);
		}
		generate_file(output_hdf, hdf);
	}

	hdf_destroy(&hdf);



	if (syslog_flag)
		closelog();

	return EXIT_SUCCESS;
}

Changes to cplanet.h.

















1
2
3



4
5
6
7
8
9
10
11
12

13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
















#ifndef CPLANET_H
#define CPLANET_H 1




#include <err.h>
#include <iconv.h>
#include <limits.h>
#include <stdarg.h>
#include <stdio.h>
#include <string.h>
#include <sys/stat.h>
#include <syslog.h>
#include <time.h>

#include <unistd.h>

/* libmrss */
#include <mrss.h>

/* clearsilver */
#include <cs/cs.h>
#include <util/neo_hdf.h>
#include <util/neo_misc.h>

#define CPLANET_VERSION "0.4"

#define CP_NAME "CPlanet.Posts.%i.Name=%s"
#define CP_FEEDNAME "CPlanet.Posts.%i.FeedName=%s"
#define CP_AUTHOR "CPlanet.Posts.%i.Author=%s"
#define CP_TITLE "CPlanet.Posts.%i.Title=%s"
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>



>
>
>









>






|
<
<







1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


40
41
42
43
44
45
46
/*
 * Copyright (c) 2010, Baptiste Daroussin
 *
 * Permission to use, copy, modify, and distribute this software for any
 * purpose with or without fee is hereby granted, provided that the above
 * copyright notice and this permission notice appear in all copies.
 *
 * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
 * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
 * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
 * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
 * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
 * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
 * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 */

#ifndef CPLANET_H
#define CPLANET_H 1

#define _XOPEN_SOURCE
#define _BSD_SOURCE
#include <errno.h>
#include <err.h>
#include <iconv.h>
#include <limits.h>
#include <stdarg.h>
#include <stdio.h>
#include <string.h>
#include <sys/stat.h>
#include <syslog.h>
#include <time.h>
#include <locale.h>
#include <unistd.h>

/* libmrss */
#include <mrss.h>

/* clearsilver */
#include <ClearSilver.h>



#define CPLANET_VERSION "0.4"

#define CP_NAME "CPlanet.Posts.%i.Name=%s"
#define CP_FEEDNAME "CPlanet.Posts.%i.FeedName=%s"
#define CP_AUTHOR "CPlanet.Posts.%i.Author=%s"
#define CP_TITLE "CPlanet.Posts.%i.Title=%s"

Changes to samples/cplanet-atom.cs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

19
20
21
22
<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
	<title type="text"><?cs var:CPlanet.Name ?></title>
	<subtitle type="text"><?cs var:CPlanet.Description ?></subtitle>
	<id><?cs var:CPlanet.URL ?>/</id>
	<link rel="self" type="text/xml" href="<?cs var:CPlanet.URL ?>/cplanet.atom" />
	<link rel="alternate" type="text/html" href="<?cs var:CPlanet.URL ?>" />
	<updated><?cs var:CPlanet.GenerationDate ?></updated>
	<author><name><?cs var:CPlanet.Name ?></name></author>
	<generator uri="http://wiki.github.com/bapt/CPlanet" version="<?cs var:CPlanet.Version ?>">CPlanet</generator>
	<?cs each:post = CPlanet.Posts ?>
	<entry>
		<title type="text"><?cs var:post.FeedName ?> >> <?cs var:post.Title ?></title>
		<?cs if:post.Author ?><author><name> <?cs var:post.Author  ?></name></author> <?cs /if ?>
		<content type="html"><![CDATA[<?cs var:post.Description ?>]]></content>
		<?cs each:tags = post.Tags ?><category term="<?cs var:tags.Tag ?>" /><?cs /each ?>
		<id><?cs var:post.Link ?></id>
		<link rel="alternate" href="<?cs var:post.Link ?>" />

		<published><?cs var:post.Date ?></published>
	</entry>
	<?cs /each ?>
</feed>





|



|




|



>
|



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
	<title type="text"><?cs var:CPlanet.Name ?></title>
	<subtitle type="text"><?cs var:CPlanet.Description ?></subtitle>
	<id><?cs var:CPlanet.URL ?>/</id>
	<link rel="self" href="<?cs var:CPlanet.URL ?>/cplanet.atom" />
	<link rel="alternate" type="text/html" href="<?cs var:CPlanet.URL ?>" />
	<updated><?cs var:CPlanet.GenerationDate ?></updated>
	<author><name><?cs var:CPlanet.Name ?></name></author>
	<generator uri="http://brokk.etoilebsd.net/projects/show/cplanet" version="<?cs var:CPlanet.Version ?>">CPlanet</generator>
	<?cs each:post = CPlanet.Posts ?>
	<entry>
		<title type="text"><?cs var:post.FeedName ?> >> <?cs var:post.Title ?></title>
		<?cs if:post.Author ?><author><name> <?cs var:post.Author  ?></name></author> <?cs /if ?>
		<content type="html"><?cs var:html_escape(post.Description) ?></content>
		<?cs each:tags = post.Tags ?><category term="<?cs var:tags.Tag ?>" /><?cs /each ?>
		<id><?cs var:post.Link ?></id>
		<link rel="alternate" href="<?cs var:post.Link ?>" />
		<updated><?cs var:post.FormatedDate ?></updated>
		<published><?cs var:post.FormatedDate ?></published>
	</entry>
	<?cs /each ?>
</feed>

Changes to samples/cplanet-opml.cs.

3
4
5
6
7
8
9
10
11
12
13
<head>
	<title><?cs var:CPlanet.Name ?></title>
	<dateCreated><?cs var:CPlanet.GenerationDate ?></dateCreated>
	<dateModified><?cs var:CPlanet.GenerationDate ?></dateModified>
</head>
<body>
	<?cs each:feed = CPlanet.Feed ?>
	<outline text="<?cs var:feed.Name ?>" htmlUrl="<?cs var:feed.Home ?>" xmlUrl="<?cs var:feed.URL ?>" />
	<?cs /each ?>
</body>
</opml>







|



3
4
5
6
7
8
9
10
11
12
13
<head>
	<title><?cs var:CPlanet.Name ?></title>
	<dateCreated><?cs var:CPlanet.GenerationDate ?></dateCreated>
	<dateModified><?cs var:CPlanet.GenerationDate ?></dateModified>
</head>
<body>
	<?cs each:feed = CPlanet.Feed ?>
	<outline text="<?cs var:feed.Name ?>" type="rss" htmlUrl="<?cs var:feed.Home ?>" xmlUrl="<?cs var:feed.URL ?>" />
	<?cs /each ?>
</body>
</opml>

Changes to samples/cplanet-rss.cs.

6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
		<description><?cs var:CPlanet.Description ?></description>
		 <language>en</language>
		 <generator>CPlanet <?cs var:CPlanet.Version ?></generator>
		 <lastBuildDate><?cs var:CPlanet.GenerationDate ?></lastBuildDate>
		 <?cs each:post = CPlanet.Posts ?>
		 <item>
			 <title><?cs var:post.FeedName ?> >> <?cs var:post.Title ?></title>
			 <description><![CDATA[<?cs var:post.Description ?>]]></description>
			 <link><?cs var:post.Link ?></link>
			 <guid><?cs var:post.Link ?></guid>
			 <?cs each:tags = post.Tags ?>
			 <category><?cs var:tags.Tag ?></category>
			 <?cs /each ?>
			 <pubDate><?cs var:post.Date ?></pubDate>
		 </item>
		 <?cs /each ?>
	</channel>
</rss>







|





|




6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
		<description><?cs var:CPlanet.Description ?></description>
		 <language>en</language>
		 <generator>CPlanet <?cs var:CPlanet.Version ?></generator>
		 <lastBuildDate><?cs var:CPlanet.GenerationDate ?></lastBuildDate>
		 <?cs each:post = CPlanet.Posts ?>
		 <item>
			 <title><?cs var:post.FeedName ?> >> <?cs var:post.Title ?></title>
			 <description><?cs var:html_escape(post.Description) ?></description>
			 <link><?cs var:post.Link ?></link>
			 <guid><?cs var:post.Link ?></guid>
			 <?cs each:tags = post.Tags ?>
			 <category><?cs var:tags.Tag ?></category>
			 <?cs /each ?>
			 <pubDate><?cs var:post.FormatedDate ?></pubDate>
		 </item>
		 <?cs /each ?>
	</channel>
</rss>

Changes to samples/cplanet.hdf.

1
2
3
4
5
6
7

8
9
10
11

12
13
14
15

16
17
18
19

20
21
22
23
24
25
26
..
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49




50
51

CPlanet {
	Name = Planet BSD Francophone
	Description = Planet regroupant sites et blogs des BSDistes francophones actifs
	Output {
		0 {
			Path = ./samples/index.html
			TemplatePath = ./samples/cplanet.cs

		}
		1 {
			Path = ./samples/cplanet.opml
			TemplatePath = ./samples/cplanet-opml.cs

		}
		2 { 
			Path = ./samples/cplanet.rss
			TemplatePath = ./samples/cplanet-rss.cs

		}
		3 {
			Path = ./samples/cplanet.atom
			TemplatePath = ./samples/cplanet-atom.cs

		}
	}
	URL =  http://planet.etoilebsd.net
	Days = 7
	Feed {
		0 {
			Name = GCU
................................................................................
			Name = Freebsd-fr 
			Home = http://www.freebsdfr.net
			URL = http://www.freebsdfr.net/?feed=atom
		}
		2 {
			Name = BSDsx
			Home = http://blog.bsdsx.fr
			URL = http://blog.bsdsx.fr/index.atom
		}
		3 {
			Name = iMil
			Home = http://imil.net
			URL = http://imil.net/wp/?feed=rss2
		}
		4 { 
			Name = Jadawin
			Home = http://feeelog.tuxaco.net
			URL = http://freelog.tuxaco.net/index.cgi/?flav=atom
		}




	}
}








>




>




>




>







 







|











>
>
>
>
|
|
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
..
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
CPlanet {
	Name = Planet BSD Francophone
	Description = Planet regroupant sites et blogs des BSDistes francophones actifs
	Output {
		0 {
			Path = ./samples/index.html
			TemplatePath = ./samples/cplanet.cs
			Type = HTML
		}
		1 {
			Path = ./samples/cplanet.opml
			TemplatePath = ./samples/cplanet-opml.cs
			Type = OPML
		}
		2 { 
			Path = ./samples/cplanet.rss
			TemplatePath = ./samples/cplanet-rss.cs
			Type = RSS
		}
		3 {
			Path = ./samples/cplanet.atom
			TemplatePath = ./samples/cplanet-atom.cs
			Type = ATOM
		}
	}
	URL =  http://planet.etoilebsd.net
	Days = 7
	Feed {
		0 {
			Name = GCU
................................................................................
			Name = Freebsd-fr 
			Home = http://www.freebsdfr.net
			URL = http://www.freebsdfr.net/?feed=atom
		}
		2 {
			Name = BSDsx
			Home = http://blog.bsdsx.fr
			URL = http://blog.bsdsx.fr/index.rss
		}
		3 {
			Name = iMil
			Home = http://imil.net
			URL = http://imil.net/wp/?feed=rss2
		}
		4 { 
			Name = Jadawin
			Home = http://feeelog.tuxaco.net
			URL = http://freelog.tuxaco.net/index.cgi/?flav=atom
		}
		5 {
			Name = Bapt
			Home = http://blog.etoilebsd.net
			URL = http://blog.etoilebsd.net/index.rss
		}
	}
}