Recess Developer Forums: Complex Boolean SQL Queries - Recess Developer Forums

Jump to content

Page 1 of 1
  • You cannot start a new topic
  • You cannot reply to this topic

Complex Boolean SQL Queries OR's and AND's should be more readily possible!

#1 User is offline   KrisJordan Icon

  • Administrator
  • Icon
  • Group: Administrators
  • Posts: 78
  • Joined: 25-August 09
  • LocationNorth Carolina, USA

Posted 17 September 2009 - 04:47 PM

Michelle brings up a point of improvement that has been needed for some time now. OR support.

View Postmichelle, on 17 September 2009 - 05:17 PM, said:

Is is possible to search multiple columns in a table for the same pattern using the mysql 'OR' keyword?
Something that would result in a statement similar to:

SELECT * FROM posts WHERE title LIKE '%keyword%' OR description LIKE '%keyword%';

Thanks!


There was once some discussion on how this could be implemented with chaining still possible, so, it would look something like:

$post->or(Cond::equal('title','%keyword%'), Cond::equal('description','%keyword%'))->orderBy('date DESC');

Who is itching to get their hands dirty with SQLBuilder?
0

#2 User is offline   christiaan Icon

  • Group: Members
  • Posts: 35
  • Joined: 06-September 09
  • LocationNijkerk, Netherlands

Posted 18 September 2009 - 01:22 AM

I've actually hit this wall also yesterday. Have been able to work around it thus far tough.

I think to keep things consistent if we decide to go this way
$post->or(Cond::equal('title','%keyword%'), Cond::equal('description','%keyword%'))->orderBy('date DESC');


The whole normal where shizzle should be changed to support proper nesting for AND as well
Imagine the following query:
SELECT * FROM posts
WHERE posted BETWEEN '2 weeks ago' AND 'today'
AND (
(title LIKE '%keyword%' AND active = 1) OR
title LIKE '%keyword%')
ORDER BY date DESC;

Resulting in a syntax like
$post->where(Cond::and(Cond::or(Cond::and(Cond::like('title','%keyword%'), Cond::equal('active', true)),
Cond::like('description','%keyword%')), Cond::between('posted', '-2 weeks', 'today'))->orderBy('date DESC');

Which I find really, really cumbersome. I like ease with which you can write the queries now.

I personally tought about the following syntax yesterday (I have really no idea if this would be possible with the current internal workings)
$post->between('posted', '-2 weeks', 'today')->
or($post->like('title', '%keyword%')->equal('active', true), $post->like('title', '%keyword%'))->orderBy('date DESC');

0

#3 User is offline   KrisJordan Icon

  • Administrator
  • Icon
  • Group: Administrators
  • Posts: 78
  • Joined: 25-August 09
  • LocationNorth Carolina, USA

Posted 18 September 2009 - 06:48 PM

View Postchristiaan, on 18 September 2009 - 02:22 AM, said:

I've actually hit this wall also yesterday. Have been able to work around it thus far tough.


Yes, this is a high priority. I am going to start moving on it soon.

View Postchristiaan, on 18 September 2009 - 02:22 AM, said:

I think to keep things consistent if we decide to go this way
$post->or(Cond::equal('title','%keyword%'), Cond::equal('description','%keyword%'))->orderBy('date DESC');


The whole normal where shizzle should be changed to support proper nesting for AND as well


The way it currently works every condition is and'ed by default. We could have two methods for flipping that switch, either andAll()/orAll(). andAll() is the default:

$post->
   orAll()->
   like('title','%keyword%')->
   like('description','%keyword%')->
   orderBy('date DESC');


This addresses simple cases but would lead to weird cases when trying to do queries over relationships:

$authorsOfKeywordWithName = 
$post->
   orAll()->
   like('title','%keyword%')->
   like('description','%keyword%')->
   author()->
     like('firstname','%name%')->
     like('lastname','%name%');
[/quote]

Would the author params be or'ed to? What would you expect?

[quote name='christiaan' date='18 September 2009 - 02:22 AM' timestamp='1253254952' post='195']
Resulting in a syntax like
[php]
$post->where(Cond::and(Cond::or(Cond::and(Cond::like('title','%keyword%'), Cond::equal('active', true)),
Cond::like('description','%keyword%')), Cond::between('posted', '-2 weeks', 'today'))->orderBy('date DESC');
[/php]
Which I find really, really cumbersome. I like ease with which you can write the queries now.
[/quote]

Agreed, this is ugly.

[quote name='christiaan' date='18 September 2009 - 02:22 AM' timestamp='1253254952' post='195']
I personally tought about the following syntax yesterday (I have really no idea if this would be possible with the current internal workings)
[php]
$post->between('posted', '-2 weeks', 'today')->
or($post->like('title', '%keyword%')->equal('active', true), $post->like('title', '%keyword%'))->orderBy('date DESC');
[/php]
[/quote]

Unfortunately you don't always have a reference to the object you're making these queries on (two specific cases: Make::a('Post')->betwee... and when you're specifying constraints on a related / joined object/table).

[quote name='Rafał Filipek' date='18 September 2009 - 07:06 PM' timestamp='1253315213' post='200']
I was thinking about all those complex find queries etc. I don't like it. As I can see You trying to keep this framework easy to use. When I'm looking at Your example Kris I can't say it is easy. IMHO 
[code]$post->or(Cond::equal('title','%keyword%'), Cond::equal('description','%keyword%'))->orderBy('date DESC');
is much more complicated than
$post->where('title LIKE "?" OR description LIKE "?"', 'foo','bar')->orderBy('date DESC');

Your example looks good for simple queries. But what about
{condition1} or {{condition2} and {condition3}}
etc.
It would be damn hard to implement so complex subconditions etc IMO.
I think we could look at Doctrine in this case : http://www.doctrine-...#select-queries
[/quote]

Having a mini-sql language is a good idea, it'll take some thought to get right though and will have to be driven by a simpler programmatic API underneath the covers.

Let's keep thinking on this.
0

#4 User is offline   Rafał Filipek Icon

  • Group: Members
  • Posts: 35
  • Joined: 28-August 09
  • LocationPoland

Posted 18 September 2009 - 07:14 PM

Ok, another idea :) Why don't we merge some simple mini-sql language ( we should keep it realy simple ) and Cond::stuff(). Cond::stuff() can be very usefull in some cases, like
Cond::in('id', array(1,4,7));
Cond::like('title', '%foo%');

etc. In those examples Cond::stuff is building some quite long string with is a part of sql query. For sure it more usefull than
$post->where('id IN(?,?,?)', 1, 4, 7);

So in fact we can do smth like.
$post->where('id? or title?', Cond::in(array(1,4,7)), Cond::like('%foo%'));

Cond::sutff don't has to know the name of the column. All it has to do is genereate some string like
Cond::in(array(1, 4, 7)); # [space]IN (1,4,7)[space]
Cond::like('%foo%'); # [space]LIKE '%foo%'[space]

that's all.
? sign is showing us where we have to put each string generated by Cond method. If we want to do smth with column name ( add ad table name ) we just have to grab by some simple RegExp a-/zA-Z0-9_/ string and the ? sign, do smth with column name and replace ? with sql part.
So now we could keep structure of compex where statements simple and use Cond::stuff etc to generate more complex parts of query
$post->where('id? and title? or (id? or (description? and foo?)))', Cond::equal(3), Cond::like('%Rafal%'), Cond::in(1,4,7), Cond::like('Filipek%'), Cond::not('bar'));

[ColumnName]? - maybe it's not the best syntax :)
Yes, in Poland we have a internet connection ;]
0

#5 User is offline   Thomaz Icon

  • Group: Members
  • Posts: 39
  • Joined: 28-August 09
  • LocationHeemskerk, The Netherlands

Posted 20 September 2009 - 08:02 AM

View PostRafał Filipek, on 19 September 2009 - 02:14 AM, said:

$post->where('id? and title? or (id? or (description? and foo?)))', Cond::equal(3), Cond::like('%Rafal%'), Cond::in(1,4,7), Cond::like('Filipek%'), Cond::not('bar'));


Well I don't think this is a very good solution, because complex queries are always quite lengthy so I think that you will loose track of all the conditions in such a query; like which Cond:: belongs to the third "?"?

Something like

$post->where('id{id} and title{title} or (id{id2} or (description{desc} and foo{foo})))', array("id"=>Cond::equal(3), "title"=>Cond::like('%Rafal%'), "id2"=>Cond::in(1,4,7), "desc"=>Cond::like('Filipek%'), "foo"=>Cond::not('bar')));


allows you to skim the query faster and keep track of conditions easier, but it has its drawbacks too..
Hoppa
0

#6 User is offline   Rafał Filipek Icon

  • Group: Members
  • Posts: 35
  • Joined: 28-August 09
  • LocationPoland

Posted 20 September 2009 - 09:48 AM

Hmmm, maybe you're right. There is only one element in your example, which I don't like. {id}, {id2}. If we want to identify each Cond in query imo we should do this like that:
$post->where(
	'{id} and {title} or ({id} or({description} and {title}))', 
	array(
		'id' => array(Cond::equal(3), Cond::in(1,4,7)),
		'title' => array(Cond::like('%Rafal%'), Cond::like('%Filipek%'),
		'description' => Cond::like('%Recess Framework%')
	);
);

Now we have some logical groups of conditions and it's easy to find specified Cond.
Yes, in Poland we have a internet connection ;]
0

#7 User is offline   KrisJordan Icon

  • Administrator
  • Icon
  • Group: Administrators
  • Posts: 78
  • Joined: 25-August 09
  • LocationNorth Carolina, USA

Posted 22 September 2009 - 11:22 AM

In learning more about jQuery this weekend I found out about how jQuery handles a similar problem with the end() function. Let me demo:

$('#myBox')
    .children('.myClass') 
       .hide()
    .end()
    .blink();


The end function returns the focus back to $("#myBox") after it was 'nested' in a more complex expression. A similar concept expressed in PHP/Recess would look like:

<?php
$post->
   or()->
     like('title','%keyword%')->
     like('description','%keyword%')->
   end()->   
   orderBy('date DESC');


If we were only concerned about one level of conditional nesting we could instead place the boolean methods between statements with no end(), or as prefixes of each selector:

<?php
$post->
   like('title','%keyword%')->
   or()->
   like('description','%keyword%')->
   orderBy('date DESC');


Or

<?php
$post->
   like('title','%keyword%')->
   orLike('description','%keyword%')->
   orderBy('date DESC');


From a readability point of view I'm a fan of the second to last option just mentioned. It's a simple solution that reads well. Perhaps nesting (which I believe is a rare case and not the case to optimize for) could be accomplished with additional syntax that's a little uglier:

<?php
$post->
   nest()->
     like('title','%keyword%')->
     or()->
     like('description','%keyword%')->
   endNest()->
   and()->
   nest()->
     lessThan('createdAt',123456)->
     or()->
     greaterThan('description','%keyword%')->
   endNest()->
   orderBy('date DESC');


When no and/or is specified the default behavior is an and() (how it currently works).

<?php
$post->like('description','%keyword%')->or()->like('title','%keyword%')->lessThan('createdAt',12345);
// Equivalent to
$post->like('description','%keyword%')->or()->like('title','%keyword%')->and()->lessThan('createdAt',12345);
// OR on all params:
$post->like('description','%keyword%')->or()->like('title','%keyword%')->or()->lessThan('createdAt',12345);
/// The following are the same:
$post->like('description','%keyword%')->like('title','%keyword%')->lessThan('createdAt',12345);
$post->like('description','%keyword%')->and()->like('title','%keyword%')->and()->lessThan('createdAt',12345);


The readability of this is growing on me.
0

#8 User is offline   Rafał Filipek Icon

  • Group: Members
  • Posts: 35
  • Joined: 28-August 09
  • LocationPoland

Posted 25 September 2009 - 09:43 AM

I like Your soluition Kris. But i think it would be hard to implement if You will have nested nests, like:
$post->
   nest()->
     like('title','%keyword%')->
     or()->
     like('description','%keyword%')->
     or()->
          nest()->
               lessThan('createdAt',123456)->
               or()->
               greaterThan('description','%keyword%')->
         endNest()->
   endNest()->
   orderBy('date DESC');

I was working on similar solution one day but i couldn't implement it :)
Yes, in Poland we have a internet connection ;]
0

#9 User is offline   spronkey Icon

  • Group: Members
  • Posts: 1
  • Joined: 16-November 09

Posted 16 November 2009 - 06:00 PM

Hi all - new to these parts, but very interested in this framework - excellent job so far guys, and with the right philosophy in my opinion!

Forgive the fact that I'm not yet 100% up to speed on the internals (working on it ;)), but I recently spent some time at my job working on our own internal framework's query and forms APIs. We quickly discovered that not only did we simply need complex boolean logic for queries, but we also needed to take further advantage of SQL when we needed to squeeze performance using switch cases, functions etc - so we basically abstracted SQL into objects. I don't know whether you folks think going this way is overkill or not, but I think it's definitely worth keeping in consideration when designing the query API. Our object model for query abstraction was similar to the following:

// top-level interfaces and AliasableExpression abstract class
interface Query {} // represents query objects as they are being built
interface IExpression {} // represents SQL expressions
interface ISelectableExpression {} // represents those that can be used in SELECT fields
interface IGroupable {} // represents groupable properties or expressions
class Alias implements IExpression, ISelectableExpression {} // represents AS with decorator pattern
abstract class AliasableExpression implements IExpression {} // represents expressions that can be aliased with AS

/* individual sql expression types
  * - SimpleValue e.g. 1, 'woot', 33.502 etc
  * - Column e.g. table.column
  * - Function e.g. REPLACE(arg,arg,arg,arg)
  * - Evaluation e.g. AliasableExpression {OPERATOR} AliasableExpression e.g. table.column = REPLACE(x,y,z)
  * - ExpressionSet with operator e.g. AliasableExpression AND AliasableExpression AND AliasableExpression
  * - Cast e.g. CAST(AliasableExpression, DATETIME)
  * - Case e.g. CASE WHEN ... THEN ... ELSE ... END
  */
class SimpleValue extends AliasableExpression implements ISelectableExpression {
   ::factory(Query $q, $value, $type);
}
class Column extends AliasableExpression implements ISelectableExpression, IGroupable {
  ::factory(Query $q, $columnOrAlias, $table = null);
}
class Function extends AliasableExpression {
  ::factory(Query $q, $functionName);
  addExpression(AliasableExpression $e);; // arguments in order - arguments can be any aliasable expression, even a switch case
}
class Evaluation extends AliasableExpression {
  ::factory(Query $q, AliasableExpression $this, $operator, AliasableExpression $that);
}
class ExpressionSet extends AliasableExpression {
  ::factory(Query $q, $logical_operator);
  addExpression(AliasableExpression $e);; // expressions imploded e.g. AE[0] $logical_operator AE[1] $logical_operator AE[2] $logical_operator AE[3]
}

/* alias class - can be used when selecting but not in other locations */
class Alias implements IExpression, ISelectableExpression {
  ::factory(IExpression $e, $alias); // $e AS $alias
}


We never got around to building the fluent query API, but we did build a (sort-of) fluent forms API using objects similar to those above. Our challenge was that we always needed access to the $form object, and had to deal with arbitrary numbers of nested containters, so a true fluent API was too serialised for our needs. The end-result we came up with produces streams as follows:

$model = new ModelObject();
$form = new Form(new URI());
$form->build(
  $form->container(
    $form->container(
      new FormModelBuilder(array('name', 'ipaddress', 'creationdate'), $model),
      new WidgetNullStateMessage(Lang::message(CREATE_A_MACHINE)
    )->template('some_form_template.tpl'),
    new FormModelBuilder(array('name2', 'someotherfield', 'creationdate'), $model)
  )
);


Reworking this into a SQL query API you can see how it could work out as something along the following lines. I'm not 100% sure I agree with you Kris regarding nesting being a rare case - I find that we often need to nest especially where using complex data views with user filtering. But I'm not sure whether this can be handled automatically outside of customised queries.

// in the following example, $post->query() uses func_get_args() to allow multiple arguments, where the and(), or(), like(), and gt() methods each return an expression
$filter = $post->filter();
$post->query(
  $post->and(
    $filter->or(
      $filter->like('title', '%keyword%'), // returns a new Like object and adds it to the set
      $filter->like('description', '%keyword%') 
    ), // returns the set with two like objects and OR operator/concatenator
    $filter->and( // the and is redundant here
      $filter->gt('creationDate', DATE_NOW)
    ), // returns the set with one filter
    $filter->gt('modificationDate', DATE_YESTERDAY), // works outside a set.
  )
)->orderBy(
   'property DESC'
); // returns the filter AND concatenator and two nested sets, and a filter applied

// outputs something along these lines
// SELECT * FROM posts WHERE ((title LIKE %keywords% OR description LIKE %keywords%) AND (creationDate > '2009-11-16 12:02:33') AND modificationDate > '2009-11-15 00:00:00'


It's not quite a 'fluent' API, but it's similar. It's effectively using a basic factory pattern on the $filter object with stuff like like(), gt(). In our framework we simply used Expression() and passed in an operator, but that is a little ugly as you have to pull the operator from somewhere.

Anyhow, I thought this might come in handy! Thoughts?
0

#10 User is offline   cuhuak Icon

  • Group: Members
  • Posts: 1
  • Joined: 17-December 09

Posted 17 December 2009 - 03:04 AM

Hi all!

I like Kris solution too.
Also I want to suggest my solution -
it is mix of propel style (using criteria) and doctrine style (using chaining) + Kris code:

$c1 = criteria($post)->
     like('title','%keyword%')->
     or()->
     like('description','%keyword%');

$c2 = criteria($post)->
     lt('createdAt',123456)->
     or()->
     like('description','%keyword%');

$post->
    criteria($c1)->
    and()->
    criteria($c2);


And what do you think about this?
0

#11 User is offline   RNewton Icon

  • Group: Members
  • Posts: 4
  • Joined: 14-April 10

Posted 14 April 2010 - 04:03 PM

Due to time crunches, we have created a simple work around for the time being, so I figured I would share. Please, no throwing of rocks or calling names, I understand this is a hack. Let me know if there is a more proper way to override the SQL statement.

Our problem was with timestamps and MySQL. The ORM produced a statement that did not work when using timestamps in the where clause. It dropped the integer in and MySQL balked at it. After a lot of trying to work around it, I just needed to write the simple SQL statement and execute it, so this is the work around I came up with. I hope it will help someone else while we wait until the full system discussed above gets implemented.

Here are the changes I made:

SqlBuilder.class.php (Changes marked with //*** SQL OVERRIDE HACK ***)
	/* SELECT */
	protected $select = '*';
	protected $selectAs = array();
	protected $joins = array();
	protected $limit;
	protected $offset;
	protected $distinct;
	protected $orderBy = array();
	protected $usingAliases = false;
	public $sqlOverride = "";  //*** SQL OVERRIDE HACK ***
	
	/**
	 * Build a SELECT SQL string from SqlBuilder's state.
	 *
	 * @return string
	 */
	public function select() {
		if ($this->sqlOverride=="") {  //*** SQL OVERRIDE HACK ***
		  $this->selectSanityCheck();
      
		  $sql = 'SELECT ' . $this->distinct . self::escapeWithTicks($this->select);
      
		  foreach($this->selectAs as $selectAs) {
		  	$sql .= ', ' . $selectAs;
		  }
		  
		  $sql .= ' FROM ' . self::escapeWithTicks($this->table);
		  
		  $sql .= $this->joinHelper();
		  
		  $sql .= $this->whereHelper();
		  
		  $sql .= $this->orderByHelper();
		  
		  $sql .= $this->rangeHelper();
		  
		  return $sql;
		}else{  //*** SQL OVERRIDE HACK ***
			return $this->sqlOverride;  //*** SQL OVERRIDE HACK ***
		}  //*** SQL OVERRIDE HACK ***
	}



PdoDataSet.class.php (Changes marked with //*** SQL OVERRIDE HACK ***)

	public function toSql() {
		if ($this->sqlBuilder->sqlOverride=="") {  //*** SQL OVERRIDE HACK ***
		  return $this->sqlBuilder->select();
		}else{  //*** SQL OVERRIDE HACK ***
			return $this->sqlBuilder->sqlOverride;  //*** SQL OVERRIDE HACK ***
		}  //*** SQL OVERRIDE HACK ***
	}

	/**
	 * Set the Sql statement override string to be returned instead of the ORM built statement
	 *
	 * @return void
	 */
	public function setSql($sqlStr) {  //*** SQL OVERRIDE HACK ***
		$this->sqlBuilder->sqlOverride = $sqlStr;  //*** SQL OVERRIDE HACK ***
	}  //*** SQL OVERRIDE HACK ***



Usage is pretty straight forward.

		$now = new DateTime();
		$startDateTime = $now->format('U') - 86400;  //Minus 24 hours
		$endDateTime = $now->format('U');
		
		//Get data for the device and date range only
		$allData = $this->Data->all();
                $this->DataSet = $allData->between('DateTime', $startDateTime, $endDateTime);

                echo $this->DataSet->toSql();
                //Output: SELECT * FROM data WHERE data.DateTime > 1271180842 AND data.DateTime < 1271266138
		
		$this->DataSet->setSql("SELECT * FROM data WHERE DateTime > timestamp('".date('Y-m-d H:i:s',$startDateTime)."') and DateTime < timestamp('".date('Y-m-d H:i:s',$endDateTime)."')");
		
		echo $this->DataSet->toSql();
                //Output: SELECT * FROM data WHERE DateTime > timestamp('2010-04-13 23:00:00') and DateTime < timestamp('2010-04-14 23:00:00')

0

#12 User is offline   HardcoreLooper Icon

  • Group: Members
  • Posts: 7
  • Joined: 11-March 10

Posted 14 April 2010 - 06:11 PM

Rob,

Thanks for posting this. I am definitely enjoying Recess, but I have to say that one of the best things I like about CodeIgniter is your ability to switch back and forth from an ORM philosophy to straight SQL when the situation merits it (like when our boss is breathing down our necks :P ).

If there's not a more correct way to run actual SQL in Recess, I hope someone can take this and expand on it.

Thanks!
Mike E.
0

#13 User is offline   msumme Icon

  • Group: Members
  • Posts: 2
  • Joined: 09-July 10

Posted 09 July 2010 - 04:24 PM

A really easy solution, that just occurred to me for "OR" would be something like this...

<?php

$modelSet = $model->all()->equal('parameter', 'value');

$modelSet2 = $model->equal('newParameter', anotherValue);

$modelSet = $modelSet->mergeResults($modelSet2);
?>
This would be better than chaining an "or" because any number of result sets can be queried and easily combined without the necessity to put or statements into the query logic. Instead, it can be in the application logic for including more than one result set as one modelset.

I'm not really sure how that would be implemented in the SqlBuilder. But I'd love to hear what people think of this idea.
0

Page 1 of 1
  • You cannot start a new topic
  • You cannot reply to this topic

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users