Discussion: View Thread

First,
My thanks to Rick Peterson, for helping arrange for the debugging of a software bug that prevented me from posting to the section. 
The bug has been resolved and  with this post  I'm again able to 1) post to the section members and 2) reply to posts from section members
This is my first post to the section since the bug was fixed.

My question here is two part -
First whether any of the section members have experience using the  R library that permits one to import conttents of a pdf file into R.  These are pdfs with lots of text.
I have successfully accomplished that import

My second question - how best to find text strings with the imported files . . And oversimplified, it seems that R imports the  separate pdfs into items in a list. So I have 39 files and I get a list with 39 items.
And finding text after selecting one item (corresponding to a specifc pdf) is where I "get stuck"

The main commands needed are to install the following two libraries
library(pdftools)
library(stringr) 

and one, creates a single working directory in R and also stores the pdfs in that same directory
then uses 
files <- list.files(pattern = "PDF$")
files
The ----->list.files,----  looks in the current /working directory for the pdfs 
My project is looking at thirty-nine separate pdf files each pdf has about 25-50 pages - of certain personal records.
These are personal and these  are not files I can share.

After this import there is an "lapply" of pdf_text
# myadobeimport is the extract text from pdf. in a list
myadobeimport <- lapply(files, pdf_text) 



It seems that each pdf is imported as  giant vector. And I have been working with substrings and character string search (regxpr, etc).
What seems to work so far is converting these to a matrix with
as.matrix(myadobeimport)

and then I search for strings by "sink()"  in items in a list and exporting to a text file ".txt", then opening and searching in the text file
sink("myadobeimport.txt")
myadobeimportmatrix <- as.matrix(myadobeimport[[3]][3])
myadobeimportmatrix
sink()

-----My last and main question, is there a shorter/faster/simpler way  for me to search for and extract specific text strings  from each pdf file?

thank you

First,
My thanks to Rick Peterson, for helping arrange for the debugging of a software bug that prevented me from posting to the section. 
The bug has been resolved and  with this post  I'm again able to 1) post to the section members and 2) reply to posts from section members
This is my first post to the section since the bug was fixed.

My question here is two part -
First whether any of the section members have experience using the  R library that permits one to import conttents of a pdf file into R.  These are pdfs with lots of text.
I have successfully accomplished that import

My second question - how best to find text strings with the imported files . . And oversimplified, it seems that R imports the  separate pdfs into items in a list. So I have 39 files and I get a list with 39 items.
And finding text after selecting one item (corresponding to a specifc pdf) is where I "get stuck"

The main commands needed are to install the following two libraries
library(pdftools)
library(stringr) 

and one, creates a single working directory in R and also stores the pdfs in that same directory
then uses 
files <- list.files(pattern = "PDF$")
files
The ----->list.files,----  looks in the current /working directory for the pdfs 
My project is looking at thirty-nine separate pdf files each pdf has about 25-50 pages - of certain personal records.
These are personal and these  are not files I can share.

After this import there is an "lapply" of pdf_text
# myadobeimport is the extract text from pdf. in a list
myadobeimport <- lapply(files, pdf_text) 



It seems that each pdf is imported as  giant vector. And I have been working with substrings and character string search (regxpr, etc).
What seems to work so far is converting these to a matrix with
as.matrix(myadobeimport)

and then I search for strings by "sink()"  in items in a list and exporting to a text file ".txt", then opening and searching in the text file
sink("myadobeimport.txt")
myadobeimportmatrix <- as.matrix(myadobeimport[[3]][3])
myadobeimportmatrix
sink()

-----My last and main question, is there a shorter/faster/simpler way  for me to search for and extract specific text strings  from each pdf file?

thank you

------------------------------
Chris Barker, Ph.D.
2023 Chair Statistical Consulting Section
Consultant and
Adjunct Associate Professor of Biostatistics
www.barkerstats.com


---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
------------------------------

First, I'll second Chris Ryan's comment on a minimal working example.  That would help.  I haven't used the pdf_text package, so references to the structure resulting from the pdf_text function are not very clear to me.

Second, I recommend using the tidytext package.  I used it about a year ago to compare word use between two sources, and it was very helpful.  @Chris Barker , I think this package will help you to complete your search pretty easily.  Given what I think I understand about your data, use the output from

  lapply(files, pdf_text)

and create a tibble or data.frame with two columns:

* filenames
* text from PDF

Given that structure, I think the following code should get you pretty close to what you need.

# Create output data where each row is a distinct sentence from a specific PDF
   pdf_unnested <- pdf_tibble %>% unnest_tokens(output = out_text, input = text_from_pdf, token = 'sentences')

# Filter data to find strings of interest
   filter(pdf_unnested, str_detect(out_text, "your string of interest") )

Splitting the pdf into different sentences seems like the approach you might need, but there are many options for the token argument that might be a better fit.  See the documentation for tidytext.  

I also found the tidytext vignettes very helpful in providing a quick start to using the package.

Hope that helps.

------------------------------
Gregory Erkens

Original Message:
Sent: 04-02-2023 16:36
From: Chris Barker
Subject: R Question - reading adobe pdf files in R

First,
My thanks to Rick Peterson, for helping arrange for the debugging of a software bug that prevented me from posting to the section. 
The bug has been resolved and  with this post  I'm again able to 1) post to the section members and 2) reply to posts from section members
This is my first post to the section since the bug was fixed.

My question here is two part -
First whether any of the section members have experience using the  R library that permits one to import conttents of a pdf file into R.  These are pdfs with lots of text.
I have successfully accomplished that import

My second question - how best to find text strings with the imported files . . And oversimplified, it seems that R imports the  separate pdfs into items in a list. So I have 39 files and I get a list with 39 items.
And finding text after selecting one item (corresponding to a specifc pdf) is where I "get stuck"

The main commands needed are to install the following two libraries
library(pdftools)
library(stringr) 

and one, creates a single working directory in R and also stores the pdfs in that same directory
then uses 
files <- list.files(pattern = "PDF$")
files
The ----->list.files,----  looks in the current /working directory for the pdfs 
My project is looking at thirty-nine separate pdf files each pdf has about 25-50 pages - of certain personal records.
These are personal and these  are not files I can share.

After this import there is an "lapply" of pdf_text
# myadobeimport is the extract text from pdf. in a list
myadobeimport <- lapply(files, pdf_text) 



It seems that each pdf is imported as  giant vector. And I have been working with substrings and character string search (regxpr, etc).
What seems to work so far is converting these to a matrix with
as.matrix(myadobeimport)

and then I search for strings by "sink()"  in items in a list and exporting to a text file ".txt", then opening and searching in the text file
sink("myadobeimport.txt")
myadobeimportmatrix <- as.matrix(myadobeimport[[3]][3])
myadobeimportmatrix
sink()

-----My last and main question, is there a shorter/faster/simpler way  for me to search for and extract specific text strings  from each pdf file?

thank you

------------------------------
Chris Barker, Ph.D.
2023 Chair Statistical Consulting Section
Consultant and
Adjunct Associate Professor of Biostatistics
www.barkerstats.com


---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
------------------------------

First,
My thanks to Rick Peterson, for helping arrange for the debugging of a software bug that prevented me from posting to the section. 
The bug has been resolved and  with this post  I'm again able to 1) post to the section members and 2) reply to posts from section members
This is my first post to the section since the bug was fixed.

My question here is two part -
First whether any of the section members have experience using the  R library that permits one to import conttents of a pdf file into R.  These are pdfs with lots of text.
I have successfully accomplished that import

My second question - how best to find text strings with the imported files . . And oversimplified, it seems that R imports the  separate pdfs into items in a list. So I have 39 files and I get a list with 39 items.
And finding text after selecting one item (corresponding to a specifc pdf) is where I "get stuck"

The main commands needed are to install the following two libraries
library(pdftools)
library(stringr) 

and one, creates a single working directory in R and also stores the pdfs in that same directory
then uses 
files <- list.files(pattern = "PDF$")
files
The ----->list.files,----  looks in the current /working directory for the pdfs 
My project is looking at thirty-nine separate pdf files each pdf has about 25-50 pages - of certain personal records.
These are personal and these  are not files I can share.

After this import there is an "lapply" of pdf_text
# myadobeimport is the extract text from pdf. in a list
myadobeimport <- lapply(files, pdf_text) 



It seems that each pdf is imported as  giant vector. And I have been working with substrings and character string search (regxpr, etc).
What seems to work so far is converting these to a matrix with
as.matrix(myadobeimport)

and then I search for strings by "sink()"  in items in a list and exporting to a text file ".txt", then opening and searching in the text file
sink("myadobeimport.txt")
myadobeimportmatrix <- as.matrix(myadobeimport[[3]][3])
myadobeimportmatrix
sink()

-----My last and main question, is there a shorter/faster/simpler way  for me to search for and extract specific text strings  from each pdf file?

thank you

First, PDF is a mess: it's a "Page Description Format," not a mechanism to convey data.  The descriptions of pages in some PDF files are extremely irregular and almost impossible to parse.

Having said that, some PDF files are tractable, especially those created by a word processor (usually).  A little experimentation suggests that when `pdf_text` reads such a file, it renders it as a list of character vectors, one per page (in order).  Newline characters "\n" separate the visible lines.  A simple call to `strsplit` will render each page as a vector of strings, one per line.

Base `R` comes with a  powerful set of text searching and processing tools embodied in its regular expression functions.  See the help page for `regular expression`.  If you're unfamiliar with the syntax of regular expressions and have more than the simplest text processing needs, then stop right here and take the time to learn it.  It's a few hours of study and practice that will pay off forever in terms of the capabilities and concepts it gives you.  If you also learn some of the underlying theory (about finite state automata) you will appreciate how incredibly efficient this technology can be.  For more about this, visit the source code page at https://github.com/laurikari/tre. 

So, the basic work flow is to read a PDF file and loop over its pages to do your processing, perhaps splitting each page into individual lines.  As a simple example, here  is the code to read a file and output all occurrences of (base-10) numbers, each accompanied by any word that might precede it (possibly separated by blanks and punctuation).  The first two lines reference and read the file; the next describes the pattern; and the last two lines match that pattern to the file's contents using `gregexec` and print out all the matches.

fn.in <- "https://example-files.online-convert.com/document/pdf/example.pdf"
S <- pdf_text(fn.in)
r <- "(^|([[:alpha:]]+[[:blank:][:punct:]]*))((([+-]?[[:digit:]]+[.]?)[[:digit:]]*)|([.][[:digit:]]+))"
op <- gregexec(r, S)
lapply(regmatches(S, op), \(x) if (length(x) > 0) x[1, ])

This processes a sample one-page PDF file available online.  Its output is

[[1]][1] "Version: 1.0"   "a 2002"         "Doe #1"         "Doe #2"         "cited 21"       "ShareAlike 3.0"

With a multiple-page file you will get a list of results, one per page.  (Any pages with no matches will simply be skipped in the output list.)

In my experience, extensive testing of any complex regular expression is always worthwhile.  This version of the pattern is a relatively untested initial attempt to illustrate the kind of search one might do to scrape a PDF file of numerical data -- so please don't rely on it implicitly!

First,
My thanks to Rick Peterson, for helping arrange for the debugging of a software bug that prevented me from posting to the section. 
The bug has been resolved and  with this post  I'm again able to 1) post to the section members and 2) reply to posts from section members
This is my first post to the section since the bug was fixed.

My question here is two part -
First whether any of the section members have experience using the  R library that permits one to import conttents of a pdf file into R.  These are pdfs with lots of text.
I have successfully accomplished that import

My second question - how best to find text strings with the imported files . . And oversimplified, it seems that R imports the  separate pdfs into items in a list. So I have 39 files and I get a list with 39 items.
And finding text after selecting one item (corresponding to a specifc pdf) is where I "get stuck"

The main commands needed are to install the following two libraries
library(pdftools)
library(stringr) 

and one, creates a single working directory in R and also stores the pdfs in that same directory
then uses 
files <- list.files(pattern = "PDF$")
files
The ----->list.files,----  looks in the current /working directory for the pdfs 
My project is looking at thirty-nine separate pdf files each pdf has about 25-50 pages - of certain personal records.
These are personal and these  are not files I can share.

After this import there is an "lapply" of pdf_text
# myadobeimport is the extract text from pdf. in a list
myadobeimport <- lapply(files, pdf_text) 



It seems that each pdf is imported as  giant vector. And I have been working with substrings and character string search (regxpr, etc).
What seems to work so far is converting these to a matrix with
as.matrix(myadobeimport)

and then I search for strings by "sink()"  in items in a list and exporting to a text file ".txt", then opening and searching in the text file
sink("myadobeimport.txt")
myadobeimportmatrix <- as.matrix(myadobeimport[[3]][3])
myadobeimportmatrix
sink()

-----My last and main question, is there a shorter/faster/simpler way  for me to search for and extract specific text strings  from each pdf file?

thank you